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Abstract 

This  thesis  develops  a  methodology  to  determine  an  optimal  policy  for  main¬ 
taining  a  satellite  constellation  that  degrades  over  time.  Previous  work  has  developed 
a  methodology  to  compute  an  optimal  replacement  policy  for  a  satellite  constellation 
in  which  satellites  were  viewed  as  binary  entities,  either  operational  or  failed.  This 
research  extends  the  previous  models  by  developing  an  optimal  maintenance  policy 
for  satellite  constellations  in  which  each  satellite  may  operate  in  a  finite  number 
of  degraded  states.  The  constellation  is  assumed  to  consist  of  a  finite  number  of 
satellites,  each  with  a  finite  number  of  functions  with  distinct  failure  mechanisms. 
Assuming  each  function  lifetime  is  exponentially  distributed,  the  stochastic  degrada¬ 
tion  process  is  modelled  as  a  discrete-time  Markov  chain.  The  degradation  process  is 
subsequently  used  to  formulate  an  optimization  problem  as  a  finite  planning  horizon 
Markov  decision  process  in  which  the  total  expected  loss  of  utility  is  minimized.  The 
maintenance  actions  considered  include  on-orbit  repairs  and  satellite  replacements, 
each  of  which  have  an  associated  level  of  risk.  Numerical  examples  are  presented 
to  illustrate  the  model,  and  a  parametric  sensitivity  analysis  is  performed  using  no¬ 
tional  data  to  determine  conditions  for  which  the  resulting  policy  consists  of  only 
replacements,  only  on-orbit  repairs,  or  mixtures  of  replacements  and  on-orbit  repairs. 
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OPTIMAL  MAINTENANCE  FOR  STOCHASTICALLY 
DEGRADING  SATELLITE  CONSTELLATIONS 


1.  Introduction 

1.1  Background 

The  world  stepped  into  the  space  age  in  1957  when  the  Soviet  Union  success¬ 
fully  launched  the  first  satellite,  Sputnik  1,  into  orbit.  Not  to  be  outdone  by  its 
Cold  War  enemy,  the  United  States  entered  the  space  arena  the  following  year  when 
Explorer  1  orbited  the  earth  for  the  first  time,  and  the  race  to  gain  command  of  the 
ultimate  high  ground  began.  Ultimately,  the  Cold  War  (and  the  space  race)  would 
come  to  an  end.  However,  its  impact  would  prove  to  have  a  significant  influence 
on  the  use  of  space  assets  for  both  military  and  commercial  uses.  Services  pro¬ 
vided  by  satellites  such  as  communications,  earth  observation,  weather  observation, 
navigation,  and  many  others  have  become  vital  to  both  public  and  private  sector 
applications.  In  a  national  defense  context,  satellites  are  vital  tools  in  intelligence 
gathering  to  help  safeguard  nations  against  enemy  threats.  Today,  multiple  satel¬ 
lites  operate  in  clusters  or  constellations  to  provide  such  services  to  end  users  on  a 
continuous  basis. 

The  costs  and  benefits  of  operating  a  satellite  constellation  are  enormous.  Each 
satellite  in  a  constellation  may  cost  up  to  hundreds  of  millions  of  dollars  to  develop, 
build,  and  launch  into  orbit.  However,  the  return  on  a  satellite  constellation  can 
also  be  extremely  high  as  billions  of  users  receive  service  from  satellite  constellations 
on  a  daily  basis.  From  a  commercial  standpoint,  this  results  in  billions  of  dollars 
in  revenue  each  year  (e.g.,  AT&T  Wireless  earned  $4.21  billion  in  revenue  in  the 
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second  quarter  of  2004  [40]).  From  a  military  standpoint,  the  returns  from  a  satellite 
constellation  result  in  high  levels  of  mission  effectiveness. 

The  enormous  potential  returns  from  a  satellite  constellation  indicate  that  any 
loss  of  capability  can  potentially  be  very  costly.  Satellites  operate  in  a  harsh  envi¬ 
ronment  and  degrade  over  time,  rendering  the  constellation  less  than  fully  capable 
as  it  ages.  This  can  lead  to  decreased  levels  of  end-user  satisfaction  which  may  result 
in  lost  revenue  or  lower  mission  effectiveness.  Therefore,  it  is  vital  to  formulate  a 
strategy  to  mitigate  the  losses  incurred  as  a  result  of  a  decrease  in  the  quality  of 
service.  In  order  to  develop  such  a  strategy,  it  is  necessary  to  determine  acceptable 
levels  of  service  loss  and  the  time  at  which  corrective  action  should  be  taken  to  im¬ 
prove  the  constellation’s  capability  level.  This  is  known  as  a  maintenance  policy.  A 
maintenance  policy  may  include  such  maintenance  actions  as  preventive  maintenance 
which  is  performed  before  a  component  fails  to  extend  the  component’s  lifetime,  or 
reactive  maintenance  which  is  performed  after  a  component  fails  to  return  it  to  a 
useful  state.  However,  due  to  the  inaccessibility  of  satellites,  preventive  maintenance 
may  not  be  a  viable  option. 

With  the  exception  of  the  Hubble  Space  Telescope,  satellite  maintenance  has 
traditionally  consisted  only  of  replacement  actions.  On-orbit  repairs  have  been  con¬ 
sidered  to  be  cost  intensive  and  too  risky  to  be  a  viable  option  for  constellation 
maintenance.  The  tragedy  of  the  Space  Shuttle  Columbia  in  2003  has  reinforced  this 
idea.  Current  technological  developments  may  provide  a  feasible  option  for  satellite 
repairs  in  the  future.  Companies  such  as  Advanced  Industrial  Science  and  Technol¬ 
ogy  (AIST)  [1]  in  Japan  are  developing  space  maintenance  robots  that  may  be  capa¬ 
ble  of  performing  on-orbit  repairs  to  satellites  at  a  fraction  of  the  cost  of  a  manned 
mission.  Thus,  a  major  question  that  arises  is:  Can  a  maintenance  policy  be  derived 
that  will  result  in  a  minimum  total  loss,  taking  into  account  satellite  maintenance 
costs  and  losses  incurred  due  to  a  less  than  fully  capable  satellite  constellation? 
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Satellite  replacements  in  the  public  sector  are  currently  driven  by  funding. 
More  specifically,  if  the  funding  for  a  new  satellite  is  available,  the  satellite  is  pur¬ 
chased.  This  can  be  a  very  costly  method  of  maintaining  a  satellite  constellation 
when  the  procurement  cost  outweighs  the  marginal  improvement  in  the  constella¬ 
tion’s  effectiveness.  Billions  of  tax  dollars  may  be  unnecessarily  spent  on  the  pur¬ 
chase  of  satellites  that  do  not  significantly  improve  mission  effectiveness.  A  more 
economically  sound  method  of  maintaining  a  satellite  constellation  is  to  purchase  a 
replacement  satellite  under  conditions  for  which  the  mission  effectiveness  improve¬ 
ment  outweighs  the  cost  of  the  replacement.  Determining  such  conditions  for  other 
general  multi-unit  systems  is  a  well-studied  topic  in  the  field  of  operations  research, 
namely  that  of  optimal  maintenance  policies. 

A  plethora  of  research  exists  concerning  optimal  maintenance  policies  for  de¬ 
grading  systems.  The  most  commonly  employed  policies  are  for  single-unit,  infinite 
planning  horizon  problems  which  generally  incorporate  three  maintenance  actions 
(do  nothing,  minimal  repair,  replace)  and  are  obtained  via  Markov  renewal  theory. 
Optimal  maintenance  policies  for  more  complex  multi-unit  systems  are  often  ob¬ 
tained  by  means  of  stochastic  dynamic  programming  using  a  discounting  factor  to 
determine  a  stationary  policy.  This  stationary  policy  prescribes  an  optimal  action 
for  every  state  in  which  the  system  may  be  observed  at  any  time.  One  drawback  to 
this  method  is  that  a  stationary  policy  does  not  always  exist.  When  the  system  is  too 
large  and  too  complex  to  be  modelled  analytically,  researchers  have  often  resorted 
to  computer  simulation  models  to  determine  approximate  maintenance  policies. 

The  United  States  Government  is  facing  the  largest  one-year  budget  deficit 
in  history  with  forecasts  as  high  as  $477  billion  in  2005.  Some  economists  believe 
that  continuing  this  spending  trend  could  have  a  catastrophic  effect  on  the  economy. 
One  of  the  largest  areas  of  government  spending  is  national  security.  However, 
due  to  the  Global  War  on  Terrorism,  the  President  and  Congress  are  unwilling  to 
sacrifice  national  security  to  reduce  the  deficit.  Even  so,  it  may  be  possible  to  reduce 
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funding  in  this  area  without  compromising  national  security.  In  order  to  accomplish 
this,  system  program  managers  must  manage  their  systems  more  efficiently,  thereby 
reducing  their  funding  requirements.  One  way  to  reduce  their  funding  requirement  is 
to  implement  optimal  maintenance  policies  for  the  systems  they  manage.  One  type 
of  system  for  which  this  would  have  a  significant  effect  is  a  satellite  constellation  dne 
to  the  high  costs  involved  and  importance  to  national  security.  While  it  will  by  no 
means  eliminate  the  budget  deficit,  implementing  an  optimal  maintenance  policy  for 
satellite  constellations  may  result  in  a  significant  reduction  in  government  spending 
while  simultaneously  achieving  national  security  objectives. 

This  research  effort  proposes  a  methodology  that  may  be  used  by  satellite 
constellation  program  managers  to  improve  the  cost  effectiveness  of  maintenance 
operations.  The  result  of  the  methodology  is  a  maintenance  policy  that  produces 
the  minimum  expected  total  cost  over  the  planning  horizon. 

1.2  Problem  Definition  and  Methodology 

We  consider  a  satellite  constellation  in  which  each  satellite  operates  at  a  finite 
number  of  capability  levels.  The  capability  levels  are  determined  by  the  number 
of  functions1  each  satellite  possesses.  Each  satellite’s  capability  degrades  over  time 
(in  the  absence  of  maintenance)  according  to  a  stochastic  degradation  process.  The 
satellites’  degradation  processes  are  assumed  to  be  stochastically  independent,  but 
the  cost  structure  for  specific  states  of  the  satellites  are  economically  dependent. 
The  capability  level  of  the  satellite  constellation  is  observed  at  discrete  points  in 
time  through  perfect  inspections.  Each  time  the  constellation  is  observed,  one  of 
three  maintenance  actions  is  chosen  for  each  satellite  (do  nothing,  on-orbit  repair, 
or  replace)  based  on  the  expected  cost  that  will  result  from  the  action  taken  and  the 
stochastic  degradation  over  the  next  inter-inspection  interval.  The  costs  associated 

1A  satellite  function  is  defined  in  Section  3.1  to  be  an  operation  that  must  be  performed  for  a 
satellite  to  successfully  carry  out  its  mission. 
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with  the  constellation  are;  a  fixed  satellite  replacement  cost,  a  state  dependent  on- 
orbit  repair  cost,  and  a  state  dependent  penalty  cost  incurred  when  the  constellation 
is  found  to  be  operating  in  a  partially  capable  state  at  any  inspection  time. 

The  primary  objective  of  this  research  is  to  provide  a  methodological  framework 
that  will  prescribe  a  provably  optimal  maintenance  policy  for  the  satellite  constella¬ 
tion  described  above.  Previous  research  in  the  area  of  optimal  satellite  constellation 
maintenance  policies  considered  satellites  as  either  “up”  or  “down”  with  the  only 
maintenance  option  being  a  satellite  replacement.  This  thesis  extends  the  previous 
research  by  allowing  satellites  to  operate  in  partially  capable  states  and  considering 
multiple  maintenance  actions.  An  optimal  maintenance  policy  for  the  satellite  con¬ 
stellation  requires  a  formally  defined  stochastic  degradation  process.  This  research 
will  consider  a  completely  observable  capability  level  that  is  a  function  of  the  oper¬ 
ational  and  non-operational  functions  of  each  satellite.  The  stochastic  degradation 
level  of  a  satellite  constellation  is  assumed  to  evolve  as  a  discrete-time  Markov  chain. 

Once  the  stochastic  degradation  process  is  defined,  an  optimization  problem 
is  formulated  to  determine  the  optimal  maintenance  policy.  The  objective  of  the 
optimization  problem  is  to  minimize  the  total  expected  loss  of  operating  a  satellite 
constellation  while  accounting  for  both  maintenance  and  penalty  costs  incurred  while 
the  constellation  is  in  partially  capable  states.  Due  to  the  complexity  and  recursive 
nature  of  the  problem,  stochastic  dynamic  programming  is  used  to  determine  the 
optimal  policy  over  a  finite  planning  horizon  by  formulating  the  the  problem  as  a 
Markov  decision  process.  A  finite  horizon  is  assumed  for  two  reasons.  First,  in 
practice  satellite  constellation  maintenance  planning  occurs  over  a  finite  horizon. 
Second,  in  a  finite  planning  horizon  at  least  one  optimal  policy  will  always  exist, 
whereas  there  is  no  guarantee  that  an  optimal  stationary  policy  exists  for  an  infinite 
horizon  problem. 
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1 . 3  Thesis  Outline 


Chapter  2  provides  a  review  of  the  pertinent  literature  with  an  emphasis  on 
optimal  maintenance  models.  Chapter  3  provides  the  formulation  of  the  stochastic 
degradation  process  for  a  satellite  constellation  followed  by  a  Markov  decision  pro¬ 
cess  formulation  which  is  subsequently  used  to  obtain  an  optimal  maintenance  policy. 
Chapter  4  provides  numerical  illustrations  using  notional  data  and  also  presents  a 
parametric  sensitivity  analysis.  Chapter  5  provides  concluding  remarks  and  sum¬ 
marizes  the  main  contributions  of  this  research,  and  recommends  the  most  fruitful 
areas  of  future  research. 
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2.  Review  of  the  Literature 

This  chapter  presents  a  review  of  the  literature  related  to  optimal  maintenance 
policies  for  stochastically  degrading  systems.  No  literature  could  be  found  pertaining 
specifically  to  degrading  satellite  constellations;  however,  much  research  does  exist  in 
closely  related  subject  areas.  An  investigation  of  current  strategies  for  maintaining 
satellite  constellations  is  presented  in  Section  2.1  with  emphasis  on  measures  of 
performance  and  budgeting  strategies.  Section  2.2  is  a  review  of  optimal  maintenance 
policies  for  single  unit  systems  and  multi-unit  systems.  Section  2.2  also  reviews 
maintenance  models  formulated  as  Markov  decision  processes. 

2.1  Satellite  Constellation  Maintenance  Planning 

Current  satellite  maintenance  planners  generally  consider  only  the  launch  of  a 
new  satellite  as  a  viable  maintenance  action,  as  on-orbit  repairs  are  considered  to 
be  both  costly  and  risky.  The  decisions  to  launch  new  satellites  typically  come  as  a 
result  of  a  resource  allocation  analysis  in  which  cost  is  considered  to  be  a  constraint 
imposed  by  the  national  defense  budget.  If  the  money  is  available,  a  satellite  may 
be  purchased,  but  the  problem  is  deciding  which  type  of  satellite  to  procure.  A 
number  of  planning  tools  aide  in  making  this  decision.  Some  of  these  tools  provide 
constellation  performance  measures  to  determine  which  constellations  are  most  in 
need  of  a  new  satellite.  Other  tools  provide  an  analysis  of  alternatives,  returning  an 
optimal  procurement  decision  that  meets  budgetary  and  performance  constraints. 

2.1.1  Performance  Measures 

Satellite  constellation  performance  measures  play  a  large  role  in  determining 
satellite  acquisition  schedules.  Two  types  of  measures  are  considered  when  determin¬ 
ing  a  replacement  policy.  The  first  is  a  measure  of  satellite  reliability,  the  probability 
that  the  satellite  survives  until  some  future  time.  The  second  is  a  measure  of  mission 
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effectiveness  which  describes  the  extent  to  which  the  constellation  meets  the  needs 
of  a  certain  mission  (e.g.,  the  percentage  of  time  an  end-user  can  receive  service). 

Until  the  late  1980’s,  it  was  widely  accepted  that  satellite  subsystem  reliabil¬ 
ity  could  be  modelled  with  an  exponential  distribution  with  failure  rates  provided 
by  MIL-HDBK-217.  Recent  research  argues  that  empirical  data  does  not  support 
a  constant  failure  rate  for  satellite  subsystems,  but  rather  a  “bathtub  curve”  fail¬ 
ure  rate  such  as  the  one  in  Figure  2.1.  In  his  1995  paper,  Hansen  [16]  provides  a 
methodology  to  determine  what  he  claims  is  a  more  accurate  and  useful  satellite 
reliability  prediction.  His  method  consists  of  performing  Monte  Carlo  simulation 
using  a  five  parameter  Weibull  distribution  to  model  the  lifetimes  of  each  satellite 
subsystem.  This  five  parameter  Weibull  distribution  exhibits  a  “bathtub  curve”  fail¬ 
ure  rate.  The  result  of  his  simulation  is  the  distribution  of  the  time  to  failure  for  the 
nth  ordered  subsystem  failure.  The  distribution  of  the  number  of  subsystem  failures 
over  a  given  interval  is  also  computed. 
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Figure  2.1  A  hypothetical  “bathtub  curve”  failure  rate  for  a  satellite. 

Adopted  with  permission  from  the  ReliaSoft  Corporation 
http:/ / www.  weibull.  com/hotwire/issue21  /hottopics21 .  htm. 
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Some  performance  measures  may  involve  both  a  reliability  measure  and  a  mis¬ 
sion  effectiveness  measure.  One  such  model  currently  used  is  presented  by  Comstock 
[11].  This  model  measures  the  effectiveness  of  a  satellite  constellation  based  on  a 
measure  defined  as  the  beginning  of  life  (BOL)  equivalence.  Each  satellite  in  the 
constellation  is  assigned  a  composite  BOL  equivalence  score.  The  BOL  equivalence 
score  takes  into  account  the  satellite’s  reliability,  duty  cycle,  and  a  subjective  band 
performance.  Each  satellite  BOL  equivalence  score  is  summed  to  determine  the  over¬ 
all  constellation  BOL  equivalence.  The  BOL  is  not  relative  to  the  performance  level 
of  the  satellite  being  measured  at  its  beginning  of  life,  but  is  actually  a  measure 
related  to  the  most  technologically  advanced  satellite  in  the  constellation.  In  other 
words,  the  BOL  equivalence  score  is  a  ratio  of  the  satellite’s  level  of  performance 
to  the  performance  level  of  the  newest  satellite  in  the  constellation.  This  allows  the 
model  to  account  for  ’’degradation”  due  to  technology  improvements  over  time. 

Another  effectiveness  measure  is  proposed  by  Wilson  [45]  who  presents  a  model 
to  measure  the  effectiveness  of  a  satellite  weather  system  that  can  be  applied  to  other 
constellations  as  well.  His  model  uses  concepts  from  multivariate  utility  theory  to 
develop  a  measure  of  effectiveness  (MOE)  to  support  decision  making.  This  MOE 
is  intended  to  measure  the  level  of  mission  accomplishment  for  a  constellation.  The 
model  divides  the  earth  into  shells  that  resemble  the  squares  formed  with  latitudinal 
and  longitudinal  grid  lines.  Each  of  these  shells  is  assigned  a  score  (MOEl  j)  which 
is  a  multiplicative  relationship  between  attributes  based  on  their  operating  level  and 
weight  assuming  utility  independence  among  attributes.  To  determine  the  overall 
MOE  he  sums  all  the  shell  MOE’s  based  on  an  assumption  of  additive  utility: 

MOE  =  EE  MOEi:j.  (2.1) 

i  j 

The  weights  used  in  the  MOE  computation  are  determined  by  a  subject  matter 
expert  (SME)  who  ranks  each  of  the  attributes  of  a  constellation  that  contribute 
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to  a  successful  mission.  Then  the  SME  assigns  each  of  the  attributes  a  rating  on 
[0, 100].  This  permits  the  creation  of  a  relative  normalized  weighting  scheme  for  the 
attributes. 

Another  currently  used  method  of  determining  the  effectiveness  of  a  satellite 
constellation  is  through  combat  simulations.  An  example  of  this  type  of  tool  is 
the  System  Effectiveness  Analysis  Simulation  (SEAS)  [37].  SEAS  is  an  agent-based 
simulation  used  to  quantify  the  contributions  of  services  provided  by  all  space  sys¬ 
tems  including  missiles,  satellites,  etc.  Measures  of  this  type  often  influence  military 
satellite  procurement  decisions  because  it  is  a  measure  of  the  combat  effectiveness 
of  current  space  assets. 

2.1.2  Satellite  Acquisition  Strategies 

The  current  strategy  of  military  space  budget  planners  is  to  spend  all  of  the 
money  they  are  allocated  in  the  most  effective  manner.  The  reasoning  behind  this  is 
that  under-utilized  resources  are  likely  to  be  retracted  at  the  end  of  the  fiscal  year, 
and  future  funding  potentially  reduced.  Decision  makers  often  fear  that  if  their 
funding  is  reduced,  they  will  not  be  able  to  meet  mission  performance  objectives  in 
the  future.  Therefore,  when  considering  the  maintenance  of  a  satellite  constellation, 
military  space  budget  planners  attempt  to  answer  the  question,  “How  much  can  we 
do  with  the  money  we  are  allotted,”  rather  than,  “How  can  we  continue  to  meet 
performance  standards  at  a  minimum  cost.” 

One  satellite  funding  allocation  tool  used  by  the  military  is  the  Space  and 
Missile  Optimization  Analysis  (SAMOA).  Brown  et  al.  [8]  describe  SAMOA  as 
a  five-component  collection  of  tools  used  to  determine  which  space  systems  will 
meet  the  requirements  of  Air  Force  Space  Command  over  a  24-year  period.  The 
SAMOA  toolkit  was  used  to  prepare  the  bi-annual  Air  Force  Space  Command’s 
Strategic  Master  Plans  in  1997  and  1999.  The  Space  Command  Optimizer  of  Util¬ 
ity  Toolkit  (SCOUT)  is  an  important  analysis  tool  within  SAMOA.  The  authors 
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describe  SCOUT  as  the  mathematical  analysis  tool  which  determines  the  optimal 
mix  of  space  systems  to  purchase  (subject  to  numerous  constraints)  via  integer  lin¬ 
ear  programming.  The  constraints  consist  of  budgetary  and  performance  measures 
which  may  or  may  not  be  elastic ,  meaning  they  may  be  violated  for  an  imposed 
penalty.  The  objective  is  to  minimize  the  summation  of  these  penalty  costs,  thereby 
meeting  mission  requirements  and  budgetary  constraints  as  much  as  possible. 

Another  tool  used  to  aide  in  determining  satellite  acquisition  policies  is  the 
Operational  Constellation  Availability  and  Reliability  Simulation  (OSCARS).  Jacobs 
et  al.  [17]  provide  a  detailed  description  of  OSCARS  which  was  developed  for  use 
by  the  Air  Force  Space  Command  Launch  Services  Office.  The  purpose  of  OSCARS 
is  to  determine  a  satellite  launch  schedule  that  will  maintain  a  specified  number 
of  operational  satellites  for  a  given  constellation.  When  determining  the  launch 
schedule,  OSCARS  takes  into  consideration  each  satellite’s  lifetime  distribution  as 
well  as  the  satellite  production  database  and  the  booster  production  database.  Any 
combination  of  satellite  lifetime  distributions  may  be  assumed  with  OSCARS  as  the 
time-dependent  distribution  of  the  number  of  operational  satellites  is  estimated  using 
Monte-Carlo  simulation. 

2.2  Optimal  Maintenance  Policies 

The  techniques  used  to  find  analytical  solutions  to  maintenance  problems  first 
appeared  in  the  literature  in  the  early  1900’s  in  actuarial  research  concerning  popu¬ 
lation  control.  By  the  1930’s,  the  industrial  revolution  had  created  a  heavy  reliance 
on  expensive  and  unreliable  industrial  equipment.  Those  researchers  studying  pop¬ 
ulation  control  problems  began  to  recognize  the  utility  of  applying  these  actuarial 
concepts  to  industrial  machinery.  Lotka  [21]  was  one  of  the  first  people  do  so  when 
he  used  the  population  growth  model  framework  to  address  the  equipment  replace¬ 
ment  problem  in  1939.  Another  early  investigation  into  the  replacement  problem 
came  in  1940  from  Preinreich  [31]  who  approached  the  problem  from  a  purely  eco- 
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nomic  standpoint.  Instead  of  considering  the  physical  degradation  of  the  machine, 
he  considered  the  economic  depreciation  of  the  machine  as  the  condition  upon  which 
to  base  replacements. 

There  are  five  basic  components  to  any  optimal  maintenance  policy  model:  the 
possible  maintenance  actions,  the  length  of  the  planning  horizon,  the  policy  class, 
the  optimization  criteria,  and  the  system  characteristics.  The  possible  maintenance 
actions  may  include  such  actions  as  minimal  repairs  which  return  the  system  to 
the  state  it  was  in  just  before  failure  or  replacements  which  return  the  system  to 
a  new  state.  The  planning  horizon  may  be  finite  or  infinite.  An  example  policy 
class  is  an  opportunistic  maintenance  policy  in  which  maintenance  is  performed 
on  multiple  components  of  a  system  in  one  maintenance  action.  The  optimization 
criteria  is  dependent  on  the  planning  horizon  and  may  be  to  minimize  the  long-run 
average  cost  per  unit  time  for  an  infinite  horizon  model,  or  it  may  be  to  minimize 
the  total  cost  for  a  finite  horizon  model.  The  system  characteristics  include  the  type 
of  failure  process  (degradation  process  or  up  down  machine)  and  the  type  of  state 
space  (discrete  or  continuous,  finite  or  infinite). 

Optimal  maintenance  policies  have  been  studied  extensively;  therefore,  it  is 
logical  to  begin  with  a  review  of  some  key  surveys.  McCall  [24]  provided  the  first 
extensive  survey  of  maintenance  models  in  his  1965  paper  which  classified  mainte¬ 
nance  scheduling  policies  by  their  common  underlying  structures.  Another  significant 
work  in  the  area  of  maintenance  policies  in  the  same  era  is  the  classic  text  by  Barlow 
and  Proschan  [4]  in  1965,  The  Mathematical  Theory  of  Reliability,  in  which  the  ba¬ 
sic  mathematical  framework  for  various  maintenance  policies  is  rigorously  reviewed. 
Pierskalla  and  Voelker  [30]  provided  another  extensive  survey  of  maintenance  models 
in  1975,  emphasizing  works  subsequent  to  the  McCall  survey.  The  1981  review  by 
Sherif  and  Smith  [36]  provides  a  brief  description  of  each  type  of  maintenance  pol¬ 
icy  and  classifies  524  major  works  by  the  subject  categories.  In  1989,  Valdez-Flores 
and  Feldman  [42]  provided  a  survey  of  maintenance  models  with  emphasis  on  the 
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single-unit  models  since  the  Pierskalla  and  Voelker  survey.  The  1991  survey  by  Clio 
and  Parlar  [10]  complements  the  Valdez-Flores  and  Feldman  survey  by  restricting 
attention  to  the  multi-unit  models  since  the  Pierskalla  and  Voelker  survey.  The  most 
recent  survey  on  this  subject  is  that  of  Wang  [44]  in  2002  in  which  he  provides  an 
updated  classification  of  policies  with  an  emphasis  on  single  unit  systems. 

Maintenance  policies  appear  in  the  literature  for  two  general  types  of  systems: 
maintenance  policies  for  single  unit  systems,  and  maintenance  policies  for  multi-unit 
systems.  Optimal  maintenance  policies  for  single-unit  systems  are  generally  found 
using  Markov  analysis  techniques  such  as  Markov  renewal  theory  to  optimize  some 
steady-state  criterion  such  as  the  long-run  average  cost  per  unit  time  or  the  long-run 
availability  of  the  system.  Optimal  maintenance  policies  for  multi-unit  systems  are 
much  more  complex  and  may  require  the  use  of  dynamic  programming  or  simulation 
for  large  problems.  Of  the  two  types  of  policies,  the  multi-unit  policies  are  most 
relevant  to  this  research;  however,  a  brief  overview  of  single-unit  policies  will  add  to 
the  understanding  of  multi-unit  policies.  Furthermore,  a  review  of  some  maintenance 
policies  found  using  Markov  decision  processes  will  also  be  provided. 

2.2.1  Single-Unit  Policies 

The  age  replacement  policy,  the  control-limit  (sometimes  referred  to  as  wear- 
limit)  policy,  and  the  inspection  policy  are  three  of  the  most  commonly  studied 
maintenance  policies  for  single-unit  systems.  These  types  of  policies  and  their  ex¬ 
tensions  are  covered  in  great  detail  in  the  literature.  Some  of  the  more  complex 
models  use  a  hybrid  policy  which  combines  two  or  more  of  these  policies  for  further 
precision. 

One  of  the  earliest  and  most  frequently  studied  categories  of  single-unit  policies 
is  the  age  replacement  policy.  This  type  of  policy  calls  for  the  instantaneous  replace¬ 
ment  of  the  system  after  operating  for  time  T  or  at  failure.  Each  time  the  system  is 
replaced,  the  stochastic  failure  process  renews  because  the  system  is  returned  to  new 
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condition.  Finding  the  optimal  age  replacement  policy  consists  of  determining  an 
optimal  operating  time  T  at  which  a  preventive  replacement  should  occur  such  that 
some  function  is  optimized.  Barlow  and  Proschan  [4:86]  show  that  for  the  infinite 
time  horizon  problem,  the  optimal  T  is  non-random.  Barlow  and  Hunter  [5]  provide 
another  optimal  solution  for  this  type  of  policy  in  which  the  limiting  efficiency  (frac¬ 
tional  amount  of  time  the  system  is  up  in  the  long  run)  is  optimized.  Nummelin  [26] 
presents  a  more  complex  variant  of  the  age  replacement  in  which  the  replacement 
cost  and  the  failure  cost  are  dependent  on  the  state  of  the  system. 

Morse  [22]  considers  a  variant  of  the  age  replacement  problem  in  which  perfect 
preventive  maintenance  or  perfect  repairs  are  performed  on  the  system  at  time  T  or 
at  failure,  respectively.  This  has  the  same  effect  as  replacing  the  system  since  the 
repairs  return  the  system  to  the  new  state.  The  difference  is  that  the  repair  times  are 
non- negligible;  the  maintenance  is  not  assumed  to  be  instantaneous.  In  that  model, 
the  objective  is  to  maximize  the  long-run  net  income  from  the  system  per  unit  time 
by  finding  the  optimal  T .  Morse  points  out  that  preventive  maintenance  is  only 
worthwhile  if  the  system  has  an  increasing  failure  rate  and  if  the  cost  to  perform 
maintenance  at  failure  exceeds  the  cost  to  perform  preventive  maintenance. 

The  control-limit  or  wear-limit  policy  is  similar  to  the  age  replacement  policy 
except  that  time  is  replaced  by  some  level  of  physical  degradation  (wear)  which  can  be 
measured.  Under  this  policy  maintenance  is  performed  when  the  wear  level  exceeds 
some  threshold  r,  or  on  failure.  When  the  system,  is  replaced,  the  stochastic  wear 
process  renews  as  in  the  age  replacement  policy  models.  Finding  the  optimal  control- 
limit  policy  consists  of  determining  the  threshold  r  such  that  the  desired  performance 
measure  is  made  optimal.  Park  [28]  shows  that  a  wear-limit  replacement  model 
in  which  the  item’s  failure  rate  is  dependent  on  some  level  of  degradation  closely 
resembles  the  age  replacement  model  when  the  long-run  total  expected  cost  rate  is 
minimized.  A  much  more  complex  wear-limit  policy  model  is  proposed  by  Su  et  al. 
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[38]  in  which  they  consider  multiple  maintenance  actions,  each  having  an  associated 
level  of  risk. 

Another  commonly  studied  category  of  single-unit  maintenance  policies  is  the 
inspection  policy.  Under  an  inspection  policy,  failures  are  only  detected  when  the 
system  is  inspected,  and  maintenance  is  performed  when  the  system  is  found  in  a 
failed  state.  A  penalty  is  incurred  for  the  time  that  a  failure  is  left  undetected.  The 
objective  of  an  inspection  policy  is  to  determine  an  inspection  schedule  such  that 
some  function  of  the  penalties,  maintenance  costs,  and  inspection  costs  is  minimized. 
Barlow  et  al.  [6]  present  the  basic  inspection  policy  model  from  which  numerous 
later  works  evolved.  In  this  paper,  only  a  single  failure  is  considered  and  the  penalty 
cost  is  dependent  on  the  time  that  the  failure  goes  undetected.  An  optimal  inter¬ 
inspection  time  is  determined  such  that  the  expected  inspection  plus  failure  costs  are 
minimized.  A  more  recent  work  concerning  inspection  policies  is  the  work  of  Grail  et 
al.  [14].  In  this  paper,  the  authors  present  results  for  a  complex  inspection  policy  in 
which  they  consider  a  wear-limit  policy  simultaneously  with  an  inspection  policy  to 
find  the  minimum  long  run  expected  maintenance  cost  per  unit  time.  Bloch-Mercier 
[7]  considers  a  similar  strategy  to  [14]  and  shows  that  under  some  assumptions,  the 
optimal  long-run  availability  is  produced  by  non-random  inter-inspection  intervals. 

2.2.2  Multi- Unit  Policies 

Replacement  policies  for  multi-unit  systems  need  only  be  studied  for  systems 
in  which  there  is  a  dependence  between  component  lifetime  distributions  or  if  the 
system  demonstrates  economic  dependence  between  components.  In  other  words,  if 
the  components  of  a  system  are  stochastically  and  economically  independent,  then 
a  single-unit  policy  can  be  applied  to  each  component  to  achieve  an  optimal  system 
maintenance  policy.  Most  of  the  literature  in  the  area  of  multi-unit  policies  considers 
the  case  of  economic  dependence.  With  regard  to  the  research  presented  in  this 


2-9 


thesis,  the  most  relevant  classes  of  multi-unit  maintenance  models  in  the  literature 
are  the  group  maintenance  policy,  and  the  opportunistic  maintenance  policy. 

The  group  maintenance  policy  is  one  of  the  most  frequently  studied  multi-unit 
policies.  In  this  type  of  policy,  all  components  of  a  multi-unit  system  are  replaced 
after  reaching  some  set  threshold  such  as  time,  number  of  component  failures,  or  cu¬ 
mulative  degradation.  If  a  component  fails  prior  to  reaching  the  established  thresh¬ 
old,  minimal  repair  or  replacement  may  be  performed  on  that  component.  The  group 
maintenance  policy,  in  some  aspects,  is  similar  to  the  control  limit  or  age  replacement 
policies  for  a  single- unit. 

One  of  the  earliest  research  efforts  with  regard  to  a  group  maintenance  pol¬ 
icy  was  the  work  of  Campbell  [9]  in  1941.  The  author  compares  two  maintenance 
policies  for  a  series  of  street  lamps.  The  first  policy  consists  of  replacing  each  lamp 
individually  as  it  fails  for  an  indefinite  period  of  time.  The  second  policy  is  a  group 
replacement  policy  in  which  all  lamps  are  replaced  at  times  {nT  :  n  =  1,2,...} 
where  T  is  the  defined  replacement  interval.  Additionally,  if  a  lamp  fails  during  this 
interval,  it  is  individually  replaced.  Campbell  assumes  that  the  lamps  each  exhibit 
an  increasing  failure  rate  and  is  able  to  find  values  of  T  for  which  it  is  economically 
beneficial  to  choose  the  group  maintenance  policy. 

Gertsbakh  [13]  proposes  a  group  maintenance  policy  for  a  system  consisting 
of  n-  in  dependent,  identical  components  having  exponentially  distributed  lifetimes. 
In  this  article,  group  maintenance  consists  of  replacing  all  failed  components  and  is 
performed  when  k  failures  first  occur.  Each  time  the  group  maintenance  is  performed, 
a  fixed  setup  cost  plus  the  replacement  cost  of  each  individual  component  is  incurred. 
Only  the  failed  components  are  replaced  because  the  lifetimes  are  assumed  to  be 
exponentially  distributed  meaning  that  replacing  a  component  that  has  not  failed  will 
not  reduce  the  chance  that  the  component  fails  in  the  next  instant.  Gertsbakh  shows 
that,  under  the  optimality  criterion  of  minimum  cost  per  unit  time  and  maximum 
return  per  unit  time,  the  previously  described  group  maintenance  policy  is  optimal. 
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A  recent  work  in  group  maintenance  policies  is  that  of  Giirlcr  and  Kaya  [15]. 
They  consider  a  system  in  which  each  component  may  be  classified  as  good,  doubt¬ 
ful,  preventive  maintenance  due  (PM  due),  and  down.  Furthermore,  each  classifi¬ 
cation  has  a  finite  number  of  degradation  levels  within  it.  They  consider  a  group 
maintenance  policy  in  which  the  system  is  replaced  when  the  number  of  doubtful 
components  reaches  a  set  threshold  and  another  component  degrades  to  PM  due  or 
down.  This  group  replacement  is  similar  to  the  control  limit  policy  of  a  single-unit, 
but  is  much  more  complex.  The  added  complexity  is  the  result  of  a  state  space 
explosion  when  the  number  of  components  and/or  the  number  of  degradation  levels 
is  large.  Because  of  this  state  space  explosion,  the  authors  do  not  provide  an  exact 
analytical  solution.  Instead,  they  propose  an  approximation  method  which  contracts 
the  state  space  in  order  to  find  a  tractable  numerical  approximation  to  the  minimum 
expected  long-run  cost. 

The  opportunistic  maintenance  policy  is  closely  related  to  the  group  main¬ 
tenance  policy.  Consider  a  system  in  which  the  cost  to  perform  maintenance  on 
multiple  units  simultaneously  is  less  than  the  sum  of  performing  maintenance  on 
each  unit  individually.  In  such  a  case,  performing  maintenance  on  one  component 
may  create  an  economically  advantageous  opportunity  to  perform  maintenance  on 
other  components.  This  generally  is  the  result  of  some  setup  cost  which  is  incurred 
each  time  maintenance  is  performed.  Most  research  concerning  opportunistic  main¬ 
tenance  policies  consider  a  /c-out-of-n  system.  That  is,  the  system  operates  if  and 
only  if  at  least  k  out  of  a  total  of  n  components  operates.  Often,  the  opportunis¬ 
tic  maintenance  policy  is  combined  with  a  variant  of  a  single-unit  policy  such  as  a 
control-limit  policy  or  an  age  maintenance  policy. 

The  grandfather  of  opportunistic  maintenance  policies  is  the  model  which  con¬ 
siders  a  system  of  only  two  components.  One  of  the  earliest  research  efforts  of  this 
type  was  contributed  in  1962  by  Radner  and  Jorgenson  [33].  In  this  work,  component 
0  and  component  1  are  independent  and  in  series  meaning  that  the  failure  of  either 
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component  causes  the  failure  of  the  system.  The  only  type  of  maintenance  action 
considered  for  either  component  is  replacement,  but  the  status  of  component  0  can 
only  be  determined  at  the  time  that  component  1  is  being  replaced.  Component  1  is 
assumed  to  have  an  exponentially  distributed  lifetime,  but  component  0  is  assumed 
to  have  a  generally  distributed  lifetime.  The  time  to  replace  both  components  at 
the  same  time  is  less  than  the  time  to  replace  each  component  individually.  The 
objective  is  to  maximize  the  expected  total  discounted  time  that  the  system  is  oper¬ 
ating  by  determining  the  best  maintenance  strategy  for  component  0.  The  solution 
method  is  a  dynamic  program  in  which  an  (■ n,N )  policy  is  found.  This  policy  is 
defined  by  three  cases: 

1.  If  component  0  has  been  operating  for  less  than  n  discrete  time  units  when 
component  1  is  replaced  due  to  failure,  then  only  replace  component  1. 

2.  If  component  0  has  been  operating  between  n  and  N  time  units  when  compo¬ 
nent  1  is  replaced  due  to  failure,  then  replace  both  components. 

3.  If  component  0  has  been  operating  for  more  than  N  time  units  and  no  failure 
has  occurred  in  component  1,  then  only  replace  component  0. 

The  two  component  problem  has  been  expanded  in  numerous  papers.  Radner 
and  Jorgensen  [34]  extended  their  previous  work  [33]  by  generalizing  their  model 
to  the  case  of  a  finite  number  of  series  components  in  which  one  component  is  not 
monitored  and  all  others  are  monitored.  Vergin  and  Scriabin  [43]  considered  a  two- 
component  and  a  three-component  system  in  which  the  components  are  economi¬ 
cally  dependent,  but  stochastically  independent.  They  use  dynamic  programming 
to  determine  an  optimal  opportunistic  preventive  maintenance  model.  A  more  re¬ 
cent  opportunistic  model  is  that  of  Pham  and  Wang  [29]  in  which  they  consider  a 
/c-out-of-n:G  system1  with  imperfect  preventive  maintenance  and  partial  failure. 

1a  fc-out-of-n:G  system  is  a  system  consisting  of  n  components,  k  of  which  must  be  good  (G) 
for  the  system  to  be  operational 
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2.2.3  Markov  Decision  Process  Maintenance  Models 


A  Markov  decision  process  is  the  specification  of  a  sequential  decision  problem 
for  a  fully  observable  environment  that  satisfies  the  Markov  assumption  under  ad¬ 
ditive  rewards.  These  types  of  problems  are  often  solved  via  (stochastic)  dynamic 
programming  to  obtain  optimal  decision  policies.  According  to  Puterman  [32:xv], 
the  use  of  Markov  decision  processes  to  solve  problems  in  numerous  disciplines  first 
appeared  in  the  1950’s.  Of  particular  interest  to  this  research  is  the  application  of 
Markov  decision  processes  to  the  maintenance  problem.  See  Section  3.4  for  further 
discussion  of  Markov  decision  processes. 

Sn  et  al.  [38]  formulated  a  Markov  decision  problem  that  seeks  the  optimal 
preventive  maintenance  policy  over  a  finite  planning  horizon  for  a  system  that  may 
occupy  a  finite  number  of  degraded  states.  There  are  a  finite  number  of  possible 
maintenance  actions  that  have  an  associated  risk  of  leaving  the  system  in  a  worse 
condition  than  that  prior  to  the  maintenance  action.  In  order  to  solve  the  problem, 
they  assume  that  the  planning  horizon,  T,  is  much  larger  than  the  time  between 
inspections,  t.  In  doing  so,  they  approximate  the  finite-horizon  problem  by  solving 
an  infinite  horizon  problem,  finding  a  stationary  policy  that  dictates  which  action 
should  be  taken  whenever  the  system  is  observed  to  be  in  each  state,  independent 
of  the  decision  epoch. 

Sumter  [39]  formulated  a  Markov  decision  process  to  find  an  optimal  replace¬ 
ment  policy  for  a  satellite  constellation  over  a  finite  time  horizon.  The  satellite  states 
are  assumed  to  be  binary  with  exponential  lifetimes;  therefore,  the  policies  found 
with  this  model  are  reactive  to  failures  (i.e.,  no  preventive  or  corrective  maintenance 
is  performed).  The  objective  was  to  minimize  the  expected  total  cost  of  maintaining 
a  satellite  constellation.  The  actions  include  replacing  a  satellite  and/or  purchasing 
a  satellite  to  use  as  a  spare,  or  do  nothing.  Because  the  purchasing  of  spares  are 
considered,  the  policy  produced  with  this  model  falls  under  the  joint  maintenance 
and  inventory  class  of  policies. 
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Moustafa  et  al.  [23]  allowed  continuous  inspections  and  non-constant  failure 
rates  by  formulating  a  semi-Markov  decision  process  to  find  an  optimal  maintenance 
policy  for  a  multi-state  semi-Markovian  deteriorating  system.  In  this  paper,  the 
objective  was  to  minimize  the  long-run  cost  rate  while  performing  continuous  in¬ 
spections  with  the  actions  being  do  nothing,  perform  minimal  repairs,  or  replace  the 
system.  An  optimal  control-limit  policy  can  be  obtained  using  the  policy-iteration 
algorithm  described  by  Puterman  [32:174], 

2.3  Research  Contributions 

Current  satellite  constellation  maintenance  problems  are  strictly  viewed  as  re¬ 
placement  problems.  As  such,  the  tools  developed  to  help  determine  constellation 
maintenance  policies  are  aimed  at  finding  the  best  time  to  acquire  and  launch  a  new 
satellite.  Various  performance  measures  are  used  to  help  determine  which  constel¬ 
lations  are  in  need  of  a  replacement  satellite  in  order  to  set  priorities  in  a  resource 
allocation  problem.  These  methods  of  determining  satellite  constellation  mainte¬ 
nance  strategies  may  result  in  the  unnecessary  expenditure  of  hundreds  of  millions 
of  dollars  for  a  satellite  that  is  not  needed. 

Optimal  maintenance  policies  have  been  studied  extensively  since  the  industrial 
revolution.  Most  optimal  maintenance  models  seek  to  determine  a  policy  in  which 
the  system  will  receive  preventive  maintenance  to  extend  the  life  of  the  system  and 
avoid  penalties  incurred  due  to  system  failures.  Policies  of  this  type  generally  rely 
on  the  assumption  that  systems  have  an  increasing  failure  rate.  However,  it  has 
been  argued  that  complex  systems  have  historically  exhibited  constant  failure  rates 
(see  for  example  [4:18]).  There  is  some  optimal  maintenance  policy  research  in  the 
literature  which  addresses  systems  with  constant  failure  rates  such  as  [13]  and  [33]; 
however,  these  models  generally  consist  of  identical  units  or  are  combined  with  units 
that  do  not  have  constant  failure  rates. 
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No  models  could  be  found  in  the  literature  concerning  the  optimal  maintenance 
of  multi-unit  system  in  which  each  component  may  exhibit  a  unique  constant  failure 
rate.  This  thesis  will  narrow  this  gap  in  the  literature  by  presenting  a  methodology 
to  determine  an  optimal  maintenance  policy  for  a  multi-unit  system  with  specific 
application  to  a  satellite  constellation.  A  constellation  is  a  multi-unit  system  com¬ 
posed  of  multi-component  satellites.  Each  satellite  component  is  assumed  to  possess 
a  constant  failure  rate,  however,  the  components  are  not  necessarily  identical.  The 
optimal  policy  will  be  obtained  by  modelling  the  entire  process  as  a  Markov  decision 
process  as  described  in  Chapter  3. 
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3.  Formal  Model  Description 

This  chapter  presents  a  formal  mathematical  model  to  determine  an  optimal 
maintenance  policy  for  degrading  satellite  constellations.  Section  3.1  provides  defi¬ 
nitions  used  throughout  and  describes  the  system  under  consideration.  Section  3.2 
describes  the  model  assumptions  employed  to  facilitate  the  mathematical  model. 
Section  3.3  presents  a  stochastic  model  for  a  satellite  constellation’s  degradation 
process  as  a  discrete-time  Markov  chain  (DTMC).  Section  3.4  presents  the  parame¬ 
ters  of  an  optimization  problem  that  can  be  solved  to  find  an  optimal  maintenance 
policy  over  a  finite  planning  horizon  while  Section  3.5  formulates  the  optimization 
problem  as  a  stochastic  dynamic  program.  Finally,  Section  3.6  provides  a  description 
of  the  evaluation  methodology  used  to  solve  the  stochastic  dynamic  program. 

3.1  System  Definitions 

Previous  work  has  considered  the  status  of  satellites  to  be  binary,  either  up  or 
down  (e.g.,  Sumter  [39]).  This  research  considers  satellites  that  may  be  degraded  but 
remain  partially  capable.  In  order  to  describe  a  constellation  with  partially  capable 
satellites,  the  following  definitions  are  needed. 

Definition  3.1  A  satellite  function  is  an  operation  that  must  be  performed  for  a 
satellite  to  successfully  carry  out  its  mission. 

Definition  3.2  A  satellite  function  is  operational  if  it  operates  as  originally  in¬ 
tended. 

Definition  3.3  A  satellite  function  is  non- operational  if  it  does  not  operate  as  orig¬ 
inally  intended. 

Consider  a  satellite  constellation  composed  of  K  satellites  each  having  a  count¬ 
able  number  of  degradation  levels.  Satellite  k  has  a  total  of  M ^  functions  that  must 
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be  operational  to  be  considered  fully  capable.  If  satellite  k  has  fewer  than  op¬ 
erational  functions,  then  the  satellite  is  considered  to  be  degraded.  The  satellites 
within  the  constellation  are  not  necessarily  identical,  the  number  of  functions  or  the 
collections  of  functions  may  be  different.  The  satellites  are  assumed  to  be  stochas¬ 
tically  independently  of  one  another;  however,  the  cost  structure  is  such  that  the 
satellites  are  economically  dependent. 

The  satellite  constellation  is  inspected  at  discrete  time  increments  (e.g.  daily, 
weekly,  or  monthly).  The  inspections  are  perfect,  meaning  that  the  true  state  of 
the  constellation  is  observed  at  each  inspection  epoch  with  probability  1.  Each  time 
the  constellation  is  observed,  a  maintenance  action  is  chosen  based  on  the  observed 
degradation  level  of  the  satellite.  In  this  context,  maintenance  actions  consist  of 
doing  nothing,  performing  on-orbit  repairs,  or  performing  a  satellite  replacement. 

3.2  Model  Assumptions 

In  order  to  represent  the  temporal  evolution  of  the  degradation  of  the  constella¬ 
tion,  it  is  necessary  to  make  a  few  simplifying  assumptions.  These  assumptions  aide 
in  reducing  the  complexity  of  the  problem  thereby  ensuring  mathematical  tractabil- 
ity.  The  first  set  of  assumptions  are  with  regard  to  individual  satellites: 

•  The  lifetime  distributions  of  satellite  functions  are  mutually  independent.  This 
means  that  the  failure  of  one  function  does  not  contribute  to  the  failure  of  any 
other  function. 

•  Satellite  functions  fail  at  a  constant  rate.  This  implies  that  the  lifetime  of 
each  satellite  function  is  exponentially  distributed.  Satellite  functions  are  very 
complex,  having  many  individual  components  that  can  fail  and  cause  the  func¬ 
tion  to  fail.  Also,  these  functions  are  expected  to  operate  for  long  periods 
of  time.  Some  communication  satellites  are  expected  to  operate  for  at  least 
10  years  [41].  As  noted  in  Barlow  and  Proschan  [4:18],  the  lifetimes  of  com- 
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plex  machines  operating  for  long  periods  tend  to  be  distributed  exponentially. 
The  impact  of  this  assumption  is  that  the  optimal  maintenance  policy  will  not 
contain  any  preventive  maintenance  (maintenance  on  an  operational  function) 
because  preventive  maintenance  will  not  decrease  the  chance  that  the  function 
becomes  non-operational  in  the  next  instant. 

The  second  set  of  assumptions  are  with  regard  to  satellite  constellations : 

•  Each  satellite  in  the  constellation  is  subject  to  degradation  independent  of 
all  other  satellites.  This  means  that  degradation  of  one  satellite  does  not 
contribute  to  the  degradation  of  any  other  satellite  in  the  constellation. 

•  If  two  or  more  satellites  have  the  same  function,  the  failure  rate  of  the  function 
is  equal  for  each  satellite. 

The  third  set  of  assumptions  are  made  with  regard  to  on-orbit  repairs : 

•  If  a  satellite  is  repaired  on-orbit,  it  is  assumed  that  the  repair  is  performed 
instantaneously.  (In  reality,  an  on-orbit  repair  may  require  months  to  initiate 
and  complete.) 

•  There  is  a  positive  probability  that  a  repair  attempt  will  be  unsuccessful.  This 
implies  that  a  repair  attempt  has  an  associated  level  of  risk  (i.e.,  an  attempted 
satellite  repair  may  not  improve  the  status  of  the  satellite). 

•  If  a  satellite  function  is  repaired  successfully,  the  function  is  assumed  to  remain 
operational  until  the  next  inspection  epoch  (i.e.,  the  function  will  not  fail  in 
the  same  time  period  in  which  it  is  repaired). 

•  If  the  satellite  function  is  repaired  successfully,  it  is  assumed  that  it  is  repaired 
to  “good  as  new”  condition. 

•  On-orbit  repair  attempts  may  only  be  made  on  functions  that  are  non-operational. 
Furthermore,  a  repair  attempt  will  not  cause  the  degradation  of  any  operational 
functions. 
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•  If  the  decision  is  made  to  perform  an  on-orbit  repair,  then  every  non-operational 
function  on  the  satellite  is  repaired. 

The  fourth  set  of  assumptions  are  made  with  regard  to  satellite  replacements : 

•  If  a  satellite  is  replaced,  it  is  assumed  that  the  replacement  is  performed  instan¬ 
taneously.  In  reality,  a  satellite  may  require  months  to  construct  and  launch 
into  orbit. 

•  If  a  satellite  is  replaced,  it  is  replaced  with  an  identical  satellite. 

•  There  is  a  positive  probability  that  a  function  will  not  survive  a  satellite  re¬ 
placement  attempt.  This  implies  that  a  replacement  attempt  also  has  an  as¬ 
sociated  level  of  risk.  The  extreme  forces  applied  to  a  satellite  during  launch, 
as  well  as  the  harsh  nature  of  the  space  environment,  are  assumed  to  induce  a 
positive  probability  that  a  function  will  arrive  on-orbit  non-operational. 

•  If  a  satellite  function  arrives  on-orbit  operational,  then  the  function  is  assumed 
to  remain  operational  until  the  next  inspection  epoch  (i.e.,  a  function  that 
survives  a  satellite  replacement  attempt  will  not  degrade  in  the  same  time 
period  as  the  replacement  attempt). 

3.3  Stochastic  Degradation  Model 

This  section  presents  the  mathematical  characterization  of  a  satellite  constella¬ 
tion’s  degradation  process.  In  order  to  determine  the  degradation  level  of  a  constella¬ 
tion,  it  is  necessary  to  consider  the  factors  that  contribute  to  the  overall  likelihood  of 
success  for  the  intended  mission.  Reeves  [35:317]  explains  that  mission  success  crite¬ 
ria  can  be  described  as  a  list  that  contains  all  of  the  events  and  operations  that  must 
occur  for  the  mission  to  be  successful.  For  the  satellite  constellation  degradation 
model,  the  operations  necessary  for  mission  success  are  the  satellite  functions.  The 
stochastic  degradation  model  is  developed  by  first  mathematically  characterizing  the 
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degradation  process  that  each  individual  satellite  undergoes  and  then  extending  the 
model  to  a  multiple  satellite  constellation. 


3.3.1  Individual  Satellite  Degradation  Process 

A  satellite  may  be  required  to  perform  several  functions  in  order  to  successfully 
perform  its  intended  mission.  Just  as  any  other  complex  machine,  its  functions  are 
subject  to  failure  due  to  degradation  over  time.  This  degradation  process  can  be 
modelled  stochastically  as  a  time-homogeneous  discrete-time  Markov  chain  (DTMC) 
requiring  the  following  assumptions  (in  addition  to  those  listed  in  Section  3.2): 

1.  There  is  a  discrete,  finite  number  of  degradation  levels  for  each  satellite. 

2.  The  probability  that  the  satellite  will  transition  between  various  degradation 
levels  between  any  two  inspection  epochs  is  the  same  for  all  inspection  epochs. 
This  property  is  known  as  time  homogeneity. 

3.  The  history  of  the  stochastic  degradation  process  is  captured  in  the  current 
degradation  level.  In  other  words,  the  probability  that  the  satellite  will  tran¬ 
sition  between  various  degradation  levels  only  depends  on  the  current  degra¬ 
dation  level  irrespective  of  the  past. 

Each  function’s  operational  status  evolves  stochastically  over  time.  For  satellite 
k  at  inspection  epoch  n,  the  status  of  function  m  is  a  binary  random  variable  defined 
by 


*)  = 


1  if  the  mth  function  of  satellite  k  is  operational  at  time  n 
0  if  the  mth  function  of  satellite  k  is  non  —  operational  at  time  n 


(3.1) 

For  example,  X1'3'1  (2)  =  1  indicates  that  at  inspection  epoch  2,  function  1  of  satellite 
3  is  observed  to  be  operational. 


Because  the  state  of  each  function  is  binary,  the  transition  probabilities  can 
be  modelled  by  reliability  functions.  In  other  words,  the  probability  that  a  function 
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survives  one  time  step  is  the  interval  reliability  of  the  subsystems  of  which  it  is  com¬ 
posed.  Define  Rm  and  Fm  as  the  interval  reliability  function  and  the  interval  lifetime 
distribution  of  satellite  function  m,  respectively.  The  interval  lifetime  distribution 
provides  the  probabilities  that  the  satellite  function  will  become  non-operational  dur¬ 
ing  the  next  time  interval  At  given  that  it  has  survived  to  the  current  time  t.  Recall 
from  Section  3.2  that  the  lifetime  of  each  function  is  distributed  exponentially.  If  we 
let  A”1  be  the  mean  lifetime  of  function  m,  then  the  lifetime  of  satellite  function  m 
is  distributed  exponentially  with  rate  parameter  Xm. 

Proposition  3.1  The  interval  lifetime  distribution  Fm  does  not  depend  on  the  time 
of  inspection.  That  is 

Fjft  +  At\t)  =  1  -  e~XmAt.  (3.2) 

Proof.  Define  Fm(t  +  At\t)  as  the  probability  that  function  m  fails 
sometime  in  the  interval  (t,  t  +  At)  given  that  it  has  survived  to  time  t.  In  other 
words,  if  the  random  lifetime  of  function  m  is  defined  as  Ym  then 


Fm(t  +  At\t)  =  P{Ym  <t  +  At\Ym  >  t}.  (3.3) 


After  unconditioning,  and  taking  the  compliment,  Equation  (3.3)  becomes 


1  -  Fm(t+  At\t) 


P{Ym  >t  +  At,  Ym>  t} 
P{Ym  >  t} 


(3.4) 


The  joint  probability  in  the  numerator  of  Equation  (3.4)  is  captured  in  the  probability 
of  the  single  event  Ym  >  t  +  At  yielding 


1  -  Fm(t  +  At\t) 


P {Ym  >  t  +  At} 
P{Ym  >  t} 


(3.5) 
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Because  the  lifetimes  are  assumed  to  be  exponentially  distributed,  Equation  (3.5)  is 
equivalent  to 

Am(t+At) 

1  -  Fm(t  +  At\t)  = - — — .  (3.6) 

g  Am L 

where  Am  is  the  failure  rate  (failures  per  period)  of  function  m.  Simplifying  Equation 
(3.6)  yields 

1  -  Fm(t  +  At\t)  = 

Therefore,  the  interval  failure  distribution  is  given  by 

Fm(t  +  At\t)  =  1  -  e~XrnAt  (3.8) 

which  is  independent  of  the  current  time  t.  ■ 

In  the  satellite  maintenance  model,  At  corresponds  to  the  time  between  deci¬ 
sion  epochs  which  is  equal  to  a  single  time  step.  Therefore, 

Fm(t  +  At\t)  =  Fm(t  +  l|f). 

If  we  let  Fm  =  Fm(t  +  l|f),  then 


e 

—  XmAt 


Xmt 


(3.7) 


Fm  =  1  -  e  A" 

which  depends  only  on  the  failure  rate  of  function  m. 


(3.9) 
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Using  Proposition  3.1,  the  interval  reliability  can  be  easily  derived.  By  defini¬ 
tion,  the  reliability  is  the  complement  of  the  lifetime  distribution.  Therefore, 

H  =  1  —  F 

1  Lrn  ±  1  m 

=  l  -  (l  -  e~Xm) 

=  e-Am.  (3.10) 

Each  satellite’s  degradation  level  is  a  row  vector  of  each  of  its  functions’  states. 
Mathematically,  the  state  of  satellite  k  having  functions  at  inspection  epoch  n 
is  given  by 

x«(»)  =  (AfV),AftO,...,vSUn)).  (3.11) 

In  other  words,  X^(n)  describes  the  random  state  of  satellite  k  at  the  nth  time 
step  as  the  vector  of  each  function’s  states.  For  example,  if  M W  =  3,  then  the 
observation  of  X(h(5)  =  (1, 1,0)  indicates  that  at  inspection  epoch  5,  satellite  1  has 
functions  1  and  2  operational  and  function  3  non-operational.  This  is  an  example  of 
a  partially  capable  satellite. 

Since  X^(n)  evolves  randomly  over  time,  the  degradation  process  of  satellite 
k  is  given  by  the  discrete-time  stochastic  process 

(XW(n)  :  n  >  0}. 

For  the  remainder  of  the  single  satellite  formulation  the  superscript  k  is  dropped 
because  there  is  only  one  satellite  in  the  constellation  being  modelled. 

A  satellite’s  level  of  degradation  evolves  over  time  and  is  characterized  by 
the  stochastic  process  (X(n)  :  n  >  0}.  Associated  with  the  stochastic  process 
(X(n)  :  n  >  0}  are  a  governing  set  of  transition  probabilities  denoted  as  p\  -v  The 
transition  probability,  pij,  is  the  probability  that  the  satellite  will  degrade  to  level 
j  given  its  history  of  degradation  and  current  state  i  by  the  next  inspection  epoch 
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where  i  and  j  are  realizations  of  the  random  vectors  X(n)  and  X(n  + 1),  respectively. 
This  transition  probability  can  be  written  as 


Pij  =  P {X(n  +  1)  =  j|X(n)  =  i,  X(n  -  1), . . . ,  X(0)}.  (3.12) 

A  satellite  transition  event  is  composed  of  the  intersection  of  each  of  its  func¬ 
tions  transition  events.  Therefore,  the  transition  probabilities  for  each  satellite’s 
degradation  process  are  the  intersection  of  the  transition  probabilities  for  each  of  its 
functions. 

By  Proposition  3.1,  Equation  (3.12)  can  be  simplified  because  each  function’s 
transition  probabilities  are  dependent  only  on  the  interval  lifetime  distribution  and 
the  current  state  of  the  function.  Furthermore,  the  interval  lifetime  distribution  is 
only  dependent  on  the  length  of  the  interval  At  which  is  1.  Therefore,  the  history  of 
the  process  is  captured  in  the  current  state  of  the  satellite.  Knowledge  of  the  state 
of  the  system  prior  to  the  current  state  is  not  necessary  to  compute  the  satellite 
transition  probabilities.  Equation  (3.12)  simplifies  to 

Pi,i  =  P  {X(n  +  1)  =  j|  X(n)  =  i}.  (3.13) 

This  is  known  as  the  Markov,  or  memoryless  property. 

Because  the  transition  probabilities  are  assumed  to  be  time- homogeneous, 
Equation  (3.13)  is  equivalent  to 


p,j  =  f{X(l)=j|X(0)  =  i}.  (3.14) 

By  the  stated  assumptions  and  Equations  (3.13)  and  (3.14),  we  can  state  that  the 
stochastic  process  (X(n)  :  n  >  0}  is  a  time-homogeneous  DTMC  with  transition 
probability  matrix  P  =  [py]. 
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In  the  case  of  the  single  satellite  model,  the  transition  probability  matrix  of 
satellite  k  is  a  2mW  x  2A/(i)  matrix.  When  the  states  are  indexed  properly,  the 
transition  probability  matrix  can  either  be  upper  or  lower  triangular. 

3.3.2  Constellation  Degradation  Process 

In  order  to  account  for  multiple  satellites  the  superscript,  k,  is  returned  to  the 
state  description  of  each  individual  satellite.  In  other  words,  the  stochastic  process 
that  characterizes  the  degradation  process  of  satellite  k  is  given  by 

(X(k)(n)  :  n  >  0}  =  { (x[k) (n) ,  1  (n) , . . .  ,x£V))  :  n  >  0}.  (3.15) 

Each  satellite  in  a  constellation  is  subject  to  the  degradation  process  described 
in  Section  3.3.1.  As  indicated  by  the  assumption  that  satellites  in  the  constellation 
are  not  necessarily  identical,  each  satellite  in  the  constellation  may  have  a  different 
number  of  elements  in  its  state  vector.  As  an  example,  take  a  constellation  with  two 
satellites.  Satellite  1  has  two  functions  and  Satellite  2  has  both  functions  of  satellite  1, 
plus  an  additional  function.  In  this  case,  K  =  2,  M ^  =  2,  and  M =  3.  Therefore, 
the  random  row  vector  X(1^(n)  would  have  two  components  and  the  random  row 
vector  X(2)(n)  would  have  three  components.  Furthermore, 

X«(n)  =  (X(1)(n),X?)(n)) 

and 

X*'2^  (n)  =  (X^  (n) ,  X^  (n) ,  Xg2)  (n) ) . 

The  multiple  satellite  model  closely  resembles  the  single  satellite  model,  how¬ 
ever,  the  constellation  model  has  a  significantly  larger  state  space.  The  overall  state 
of  the  constellation  is  a  row  vector  of  the  states  of  each  of  the  satellites  within  the 
constellation.  If  there  are  K  satellites  in  the  constellation,  then  at  inspection  epoch 
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n  the  state  of  the  constellation  is  given  by 


X(n)  =  (X(1)(n),X(2)(n),...,Xw(n)).  (3.16) 

Using  the  previous  example,  the  observation 

X(5)  =  ((1,1),  (0,1,1)) 

indicates  that  at  inspection  epoch  5,  satellite  1  is  fully  capable  and  satellite  2  is 
partially  capable. 

Since  X(n)  evolves  stochastically  over  time,  the  overall  degradation  process  of 
the  constellation  is  given  by  the  discrete  time  stochastic  process 

(X(n)  :  n  >  0}. 

Furthermore,  because  each  satellite’s  degradation  process  is  a  time-homogeneous 
DTMC  and  each  satellite  is  stochastically  independent,  the  constellation  degradation 
process  (X(n)  :  n  >  0}  is  also  a  time-homogeneous  DTMC. 

The  transition  probabilities  for  the  multiple  satellite  model  are  similar  to  those 
in  the  single  satellite  model.  However,  the  transition  probability  matrix  is  much 
larger  due  to  the  additional  states.  In  the  case  of  the  multiple  satellite  model,  the 
transition  probability  matrix  is  a  2 '£k=iM(k)  x  2 FT=iM(fc)  matrix.  For  example,  if  a 
constellation  consists  of  two  satellites  with  ilU1'  =  2  and  ibC2)  =  3  then  the  transition 
probability  matrix  contains  22+3  =  32  rows  and  columns.  As  in  the  individual 
satellite  formulation,  when  the  states  are  indexed  properly,  the  transition  probability 
matrix  can  be  either  upper  or  lower  triangular. 

A  constellation  transition  event  is  composed  of  the  intersection  of  each  of  its 
satellites  transition  events.  Therefore,  the  transition  probabilities  for  the  constella¬ 
tion  degradation  process  are  the  intersection  of  the  transition  probabilities  for  each  of 
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its  satellites.  But  since  the  satellites  are  stochastically  independent,  the  intersection 
can  be  expressed  as  a  product. 

The  case  of  a  one  satellite  constellation,  K  —  1,  is  a  special  case  of  the  con¬ 
stellation  formulation  and  the  model  becomes  identical  to  that  of  section  3.3.1.  As 
an  illustration  of  this,  take  a  one  satellite  constellation  in  which  the  satellite  has  two 
functions.  Then  the  constellation’s  random  state  at  inspection  epoch  n  is  given  by 

X(n)  =  X(1)(n). 


But 

X<‘>(n)  =  (.yynJ.AfV)). 


Therefore, 

X(n)  =  (Af>(n),Af  >(«)) 

which  is  identical  to  the  single  satellite  formulation. 

To  illustrate  the  stochastic  degradation  process,  consider  a  two  satellite  con¬ 
stellation  in  which  Satellite  1  has  2  functions  and  Satellite  2  has  1  function  which  is 
different  than  either  function  of  Satellite  1.  In  other  words,  K  =  2,  M^1)  =  2,  and 
M(2)  —  i.  Therefore,  there  are  22+1  =  8  total  degradation  levels.  The  enumerated 
degradation  levels  for  this  constellation  are  given  in  Table  3.1. 

Table  3.1  Two  satellite  state  space. 


State 

Degradation  Level 

Sl 

((1,1),  (1)) 

S2 

((1,1),  (0)) 

S3 

((1,0),  (1)) 

s4 

((1.0), (0)) 

S5 

((0,1),  (1)) 

S6 

((0.1),  (0)) 

S7 

((0,0), (1)) 

S8 

((0,0),  (0)) 
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A  possible  sample  path  of  (X(n)  :  n  >  0}  is  given  in  Figure  3.1.  When  no 
maintenance  actions  are  considered,  the  constellation  cannot  transition  to  a  less 
degraded  state  as  indicated  by  the  non- increasing  sample  path. 

X<2>(«) 

(1,1,1) - 

(1,1,0)  - 

(1,0,1)-  1 - ; 

(1,0,0)-  i - , 

(0,1,1)  - 
(0,1,0)  _ 

(0,0,1)  _ 

(0,0,0)  _  I _ 


0  1  2  3  4  5  6  7 

Figure  3.1  Possible  sample  path  for  (X(n)  :  n  >  0} 

The  transition  probability  matrix  is  an  8  x  8  matrix.  There  are  three  different 
functions  in  the  constellation,  therefore,  there  are  three  different  values  for  each  of 
Rrn  and  Fm.  For  function  1  of  satellite  1,  m  —  1.  For  function  2  of  satellite  1,  m  —  2. 
For  the  function  on  satellite  2,  m  =  3.  The  transition  probability  matrix  is  easily 
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determined  to  be 


R1R2R3  R.\R,2 F3  R 1  F'2 R‘i  R 1 F2 F?j  F\  R2 R-3  F\  R-2 F‘,i  F\  F2 R;s  F\  F2 F$ 

0  R\R2  0  R\  F2  0  F\  R‘2  0  F\  F2 

0  0  R1R3  R1F3  0  0  F1R3  F\  F'2, 

0  0  0  R1  0  0  0  F1 

0  0  0  0  R2R3  R2F3  F2R3  F2F3 

0  0  0  0  0  r2  0  f2 

0  0  0  0  0  0  r3  f3 

0  0  0  0  0  0  0  1 

3.4  Optimization  Problem 

Now  that  the  degradation  process  for  a  satellite  constellation  has  been  de¬ 
scribed,  we  can  formulate  an  optimization  problem  that  can  be  solved  to  find  an 
optimal  maintenance  policy  for  the  constellation.  The  first  step  in  formulating  an 
optimization  problem  is  to  define  the  goal  of  optimization  problem  in  terms  of  an  ob¬ 
jective  function.  The  second  step  is  to  determine  the  decision  variables,  the  variables 
that  can  be  changed  to  affect  the  objective  function.  The  third  step  is  to  identify 
any  constraints  imposed  on  the  values  of  the  decision  variables.  Finally,  a  suitable 
optimization  technique  must  be  chosen. 

3-4-1  Objective  Function 

The  goal  of  the  optimization  problem  is  to  find  the  best  maintenance  policy 
over  a  finite  planning  horizon.  In  order  to  determine  which  policy  is  “best,”  a  value 
must  be  assigned  to  each  policy  so  that  they  can  be  represented  in  an  ordered  set. 
The  objective  function  defines  how  values  are  assigned  to  each  policy  and  determines 
whether  to  seek  the  maximum  or  the  minimum  of  this  set  (assuming  a  finite  number 
of  policies).  Some  of  the  candidate  objectives  considered  in  this  research  were 
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•  minimizing  the  total  expected  cost  of  maintaining  the  constellation, 

•  maximizing  mission  effectiveness, 

•  maximizing  profit, 

•  minimizing  loss  of  service,  and 

•  minimizing  the  total  loss  of  utility. 

After  careful  consideration,  the  objective  of  minimizing  the  expected  total  loss 
of  utility  was  chosen.  This  objective  was  chosen  because  there  are  two  types  of  costs 
associated  with  maintaining  a  satellite  constellation.  The  first  cost  is  a  deterministic 
monetary  cost  incurred  when  a  maintenance  action  is  chosen.  For  example,  when 
a  satellite  is  replaced,  there  is  a  known  procurement  cost.  The  second  cost  is  a 
penalty  cost,  possibly  monetary  in  the  private  sector,  or  some  other  unit  of  mission 
effectiveness  in  the  military  sector.  For  example,  when  a  commercial  communication 
satellite  is  operating  in  a  partially  capable  state,  a  monetary  value  associated  with 
loss  of  customer  goodwill  might  be  incurred;  but  in  the  military  sector,  the  cost 
might  be  lost  lives  because  of  a  communications  error  on  the  battlefield.  When 
considering  the  business  application,  the  utility  is  measured  on  the  real  line  in  units 
of  dollars.  When  considering  the  military  application,  the  two  types  of  cost  are  in 
different  units  and  must  therefore  be  transformed  to  some  common  unit  of  utility. 

3-4-2  Decision  Variables 

The  decision  variables  for  this  problem  are  the  maintenance  actions  to  take  at 
decision  epochs  when  the  constellation  is  observed  to  be  in  a  certain  state.  In  other 
words,  which  maintenance  action  should  be  taken  when  the  constellation  is  observed 
to  be  in  state  s  at  time  n  for  every  possible  s  and  n? 

The  feasible  maintenance  actions  for  this  model  are  do  nothing,  perform  one 
or  more  on-orbit  repairs,  or  replace  one  or  more  satellites.  Previous  constellation 
maintenance  models  have  considered  only  replacements  as  a  feasible  means  of  repairs 
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due  to  the  high  cost  involved  with  performing  an  on-orbit  repair  (e.g.  Sumter  [39]). 
However,  on-orbit  repairs  may  become  a  more  viable  option  with  the  development 
of  satellite  repair  robots  which  could  repair  satellites  at  a  fraction  of  the  cost  of  a 
manned  mission  [27]. 

3-4-3  Solution  Strategy 

We  formulate  the  problem  as  a  Markov  decision  process  and  solve  it  via  stochas¬ 
tic  dynamic  programming.  This  formulation  was  chosen  because  the  maintenance 
policy  it  produces  can  be  mathematically  proven  to  be  optimal.  An  additional  reason 
that  this  technique  was  selected  is  that  it  produces  a  policy  that  is  easy  to  interpret. 
An  illustration  of  the  ease  of  interpretability  is  provided  in  Chapter  4. 

According  to  Pnterman  [32:17],  “a  Markov  decision  process  model  consists  of 
five  elements:  decision  epochs,  states,  actions,  transition  probabilities,  and  rewards.” 
A  clarification  of  each  of  the  five  elements  of  a  MDP  is  provided  below. 

Definition  3.4  A  decision  epoch  is  a  point  in  time  at  which  the  system  is  inspected 
and  a  decision  is  made  to  affect  the  system  based  on  that  observation.  The  set  of 
decision  epochs  can  be  finite  or  infinite;  discrete  or  a  continuum. 

Definition  3.5  At  each  decision  epoch,  the  state  of  the  system  is  what  is  observed 
and  an  allowable  action  is  chosen  based  on  that  observed  state.  According  to  Puter- 
man  [32:18]  the  state  and  action  sets  “may  each  either  be 

1.  arbitrary  finite  sets; 

2.  arbitrary  countably  infinite  sets; 

3.  compact  subsets  of  finite  dimensional  Euclidean  space;  or 

4 ■  non-empty  Borel  subsets  of  complete,  separable  metric  spaces.” 

Definition  3.6  Each  time  an  action  is  chosen,  the  system  evolves  according  to 
a  probability  distribution  determined  by  the  observed  state  of  the  system  and  the 
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action  chosen.  The  collection  of  these  probability  distributions  is  known  as  the 
transition  probabilities.  This  set  also  includes  the  effect  transitions,  or  the  satellite 
degradation  transition  matrix. 

Definition  3.7  A  reward  is  received  each  time  an  action  is  chosen.  The  reward 
takes  on  positive  values  for  “income”  or  “gain”,  and  negative  values  for  “cost”  or 
“loss”.  According  to  Puterman  [32:20],  the  “reward  might  be 

1.  a  lump  sum  received  at  a  fixed  or  random  time  prior  to  the  next  decision  epoch; 

2.  accrued  continuously  throughout  the  current  period; 

3.  a  random  quantity  that  depends  on  the  system  state  at  the  subsequent  decision 
epoch;  or 

4-  a  combination  of  the  above.” 

In  this  application,  a  MDP  is  formulated  to  determine  a  policy  that  minimizes 
the  loss  associated  with  maintaining  a  degrading  satellite  constellation.  A  policy  is 
denoted  by  a  vector  7 r  =  ( di,d2 , .  ■  ■ ,  c?jv— i)  such  that  dn  indicates  the  decision  rule 
for  decision  epoch  n.  The  decision  rule  for  decision  epoch  n  provides  an  action  to 
take  for  each  state  in  which  the  system  may  be  observed  at  time  n.  (Note  that 
there  is  no  decision  made  at  the  final  epoch,  N.)  An  optimal  policy  is  denoted 
by  7r*  =  where  d*  indicates  the  optimal  decision  rule  for  decision 

epoch  n.  In  other  words,  for  each  n,  d*  is  the  vector  of  optimal  actions  to  take  for 
every  possible  state  in  which  the  system  could  be  observed  at  decision  epoch  n.  For 
example,  if  d%  =  (1, 1,4)  then,  if  at  decision  epoch  3  the  system  is  observed  to  be  in 
states  1  or  2,  action  1  is  optimal;  if  at  decision  epoch  3  the  system  is  observed  to  be 
in  state  3,  action  4  is  optimal. 

To  illustrate  the  policy  produced  by  a  Markov  decision  process,  consider  a  single 
satellite  with  two  functions.  The  planning  horizon  is  10  periods  and  the  allowable 
actions  are: 
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1.  -  do  nothing, 

2.  -  repair  the  satellite,  or 

3.  -  replace  the  satellite. 


Then  an  optimal  policy  may  be  as  given  in  Table  3.2.  If  the  current  epoch  is  n  =  3, 
then  the  optimal  decision  rule  is  d3  =  (1,3,  2,  3).  This  means  that  at  time  n  =  3,  if 
the  satellite  is  observed  to  be  in  state  Si ,  then  the  optimal  action  to  take  is  to  do 
nothing.  If  the  satellite  is  observed  to  be  in  states  S2  or  S4  then  the  optimal  action  is 
to  replace  the  satellite.  If  the  satellite  is  observed  to  be  in  state  s3  then  the  optimal 
action  is  to  perform  an  on-orbit  repair. 


Table  3.2  Optimal  policy  example. 


State 

Decision  Epoch  (n) 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

si  =  (1,1) 

1 

1 

1 

1 

1 

1 

1 

1 

1 

N/A 

52  =  (1,0) 

3 

3 

3 

1 

1 

1 

1 

1 

1 

N/A 

53  =  (0, 1) 

2 

2 

2 

2 

2 

2 

1 

1 

1 

N/A 

54  =  (0,0) 

3 

3 

3 

3 

3 

3 

3 

3 

3 

N/A 

3.5  Formal  Markov  Decision  Process  Formulation 

In  formulating  the  satellite  maintenance  model  as  a  Markov  decision  process, 
we  assume  that  the  action  sets  are  stationary.  This  means  that  the  set  of  allowable 
actions  at  decision  epoch  1  is  the  same  as  the  set  of  allowable  actions  at  decision 
epoch  n  for  every  n.  The  elements  of  a  MDP  with  regard  to  the  satellite  maintenance 
model  follow. 

3.5.1  Decision  Epochs 

In  the  case  of  the  infinite  horizon  problem,  a  stationary  policy  only  exists  if 
the  underlying  Markov  chain  is  ergodic  (irreducible,  recurrent,  and  aperiodic).  This 
condition  is  not  guaranteed  to  hold  for  the  satellite  maintenance  model.  When 
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considering  a  finite  planning  horizon,  at  least  one  optimal  policy  will  always  exist. 
Therefore,  the  planning  horizon  for  the  satellite  constellation  maintenance  model  is 
finite.  Also,  inspections  of  a  constellation  occur  at  constant  intervals  to  determine 
its  operational  status.  Therefore,  the  time  steps  are  discrete.  The  set  of  decision 
epochs  is  given  by 

N=  {1,2,...,  AT},  N  <  oo.  (3.17) 

In  other  words,  the  inspections  and  decisions  are  made  at  discrete  intervals.  They 
could  represent  inspections  that  occur  daily,  quarterly,  annually,  etc. 

3.5.2  State  Space 

The  state  space  for  each  individual  satellite  in  the  MDP  model  is  composed 
of  a  finite  number  of  degradation  levels  as  described  in  section  3.3.1.  The  random 
state  of  satellite  k  of  the  constellation  is  given  by 

X(fc)(n)  =  (Af  }(n),xf  (n), . . .  ,x£\n)).  (3.18) 

The  state  space  for  the  entire  constellation  is  composed  of  a  finite  number  of 
degradation  levels  as  described  in  Section  3.3.2.  The  random  state  of  the  constella¬ 
tion  at  decision  epoch  n  can  be  written  as 

X(n)  =  (X(1)(n),  X(2)(n), . . . ,  X(K)(n))  (3.19) 

where  X®(n)  is  the  random  state  of  satellite  k  at  inspection  epoch  n. 

The  total  number  of  states,  or  the  cardinality  of  satellite  k’ s  state  space  S^.  is 

\Sk\  =  2M(k)  (3.20) 
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and  the  cardinality  of  the  state  space  for  the  entire  constellation,  S,  is 


51  =22?.."'*', 


(3.21) 


3.5.3  Actions 

Satellite  k  has  an  associated  action  set  denoted  by  A(k)  that  is  an  indexed  listing 
of  all  possible  actions  that  can  be  taken  for  satellite  k.  Some  actions  in  this  set  may 
not  be  allowed  when  satellite  k  is  observed  to  be  in  certain  states.  Therefore,  the 
action  set  for  every  state  s  is  a  subset  of  A ^  denoted  by  Aik> .  These  satellite  action 
subsets  are  the  same  for  every  decision  epoch  due  to  the  assumption  of  stationary 
action  sets.  If  s  denotes  the  observed  state  of  satellite  k  and  i  denotes  the  index  of 
the  action,  then  a  specific  action  for  satellite  k  in  state  s  is  given  by 


For  example,  action  2  for  satellite  5  when  observed  to  be  in  state  si  is  denoted  by 

(5) 

<,2- 

The  action  set  for  the  entire  constellation,  denoted  by  A ,  can  be  described  as 
an  indexed  listing  of  all  possible  combinations  of  individual  satellite  actions.  As  in 
the  individual  satellite  action  sets,  not  all  actions  are  allowable  for  every  state  the 
constellation  can  occupy.  Therefore,  the  action  set  for  every  constellation  state  s'  is 
a  subset  of  A  denoted  by  As>-  If  s'  denotes  the  observed  state  of  the  constellation 
and  i'  denotes  the  index  of  the  constellation  action,  then  a  specific  action  for  the 
constellation  in  state  s'  is  given  by  the  K -dimensional  row  vector 


£  As  —  {(ci^ 


(1) 
i  ? 


{K)s 


t  ® s,i 


e  4fc)} 


(3.22) 
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As  an  illustration,  take  a  constellation  that  consists  of  two  satellites  with  two  possible 
actions  for  each  satellite,  then 


/  (1)  (2)  x 

*4,2'  = 

would  indicate  that  constellation  action  2r  when  the  constellation  is  in  state  s'9  means 
to  take  action  1  for  the  first  satellite  when  in  that  satellite’s  state  S3  and  action  2 
for  the  second  satellite  when  in  that  satellite’s  state  si. 

3.5.4  Transition  Probabilities 

The  following  parameters  are  necessary  to  determine  the  transition  probabili¬ 
ties: 


•  Rm  and  Fm  are  respectively  the  interval  reliability  and  the  interval  lifetime 
distribution  of  satellite  function  m.  The  lifetime  of  satellite  function  m  is 
assumed  to  be  distributed  exponentially  with  rate  parameter  Am. 

•  Hm  is  the  Bernoulli  probability  mass  function  (p.m.f.)  associated  with  the 
probability  that  function  m  is  successfully  repaired. 

•  Gm  is  the  Bernoulli  p.m.f.  associated  with  the  probability  that  function  m 
survives  a  satellite  replacement  attempt. 

As  in  Su  et  al.  (2000)  [38],  Hrn  and  Grn  imply  the  possibility  of  imperfect  mainte¬ 
nance.  This  means  that  a  measure  of  risk  is  associated  with  choosing  a  maintenance 
action  (i.e.,  there  is  a  possibility  that  the  system  will  not  improve  as  a  result  of 
performing  the  maintenance  action). 

In  order  for  the  constellation  to  transition  from  one  state  to  another,  each 
satellite  in  the  constellation  must  undergo  a  transition,  even  if  the  transition  is  from 
one  state  to  the  same  state.  Therefore,  the  transitions  of  each  individual  satellite  are 
discussed  first  followed  by  a  discussion  of  the  transitions  of  the  entire  constellation. 
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Let  h(n)  denote  the  history  of  the  states  and  actions  for  an  individual  satellite 
up  to  n  —  1.  Mathematically,  the  history  is  given  by 

h(n)  =  a(j ))  :  1  <  j  <  n  —  1} 

where  each  s (j )  is  a  realization  of  the  random  vector  Denote  pn'1  {j\s(n),  a{n),  h{n)} 

as  the  probability  at  decision  epoch  n  that  satellite  k  will  transition  to  state  j  G 
given  its  current  state  s(n),  the  action  taken  a(n),  and  the  history  h(n).  For  example 

P?)  {Sl(4)|s4(3),  a3(3),  (si(2),  m(2),  s3(l),  ai(l))} 

is  the  probability  at  decision  epoch  3  that  satellite  2  will  transition  to  state  si  given 
the  current  state  S4,  the  action  taken  as,  and  the  history  h( 3)  (state  Si  and  action 
a  1  at  n  —  1,  state  s3  and  action  a 1  at  n  —  2). 

By  the  Markov  assumption,  the  probability  at  decision  epoch  n  +  1  that  the 
constellation  will  transition  to  state  j  G  S  given  the  current  state  s(n),  the  current 
action  a(n),  and  the  history  h(n),  is  dependent  only  on  the  most  recent  observed 
level  of  degradation  and  the  action  taken.  That  is, 

Pn  ]  {j\s(n),  a(n),  h(n)j  =  p ^  (j|s(n),  a(n)}  (3.23) 

where  n  —  1,  2, . . . ,  N  and  k  =  1,  2, ... ,  K . 

As  in  the  stochastic  degradation  model,  the  constellation  transition  events  are 
composed  of  the  intersection  of  each  of  its  satellites  transition  events.  Therefore, 
the  transition  probabilities  for  the  constellation  are  the  intersection  of  the  transition 
probabilities  for  each  of  its  satellites.  But  since  the  satellites  are  independent,  the 
intersection  can  be  expressed  as  a  product.  Proposition  3.2  clarifies  this  point. 

Proposition  3.2  Suppose  that  at  decision  epoch  n,  a  satellite  constellation  is  ob¬ 
served  to  be  in  state  s  G  S.  Then  the  probability  at  decision  epoch  n  +  1  that  the 
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constellation  will  transition  to  state  j  E  S  given  the  current  state  s  and  the  current 
action  vector  a S)i  is  given  by 


K 


Pi 


{j|s,aSji}  =  Y[pok)  {j(fc)|s(fc),asW>J 


(3.24) 


k= 1 


where  s^k\j^  G  S(k\  k  —  1, . . . ,  K. 

Proof.  By  definition, 

Pn  01 S,  as,i}  =  Pn  0'(1),  j{2\  ■  •  •  ,j{K)\s{1),  S(2),  .  .  .  ,  S(K\  0,(1)^,  0,(2) >i,  .  .  .  ,  as(K)  J  . 

(3.25) 

By  the  assumption  that  satellites  are  subject  to  degradation  independently  of  each 
other,  the  joint  probability  in  Equation  (3.25)  is  equivalent  to 

Pn  Ols,aS)i}  =Pn]  0'(1)|s(1)>  asW,i}XPn^  {j^l^ ,  aSV  ),i}x...Xp^  {j{K)  |sW,  a, S(K) 

(3.26) 

Since  each  satellite’s  degradation  process  is  assumed  to  have  stationary  transition 
probabilities  in  discrete  time,  we  can  state  that 

Pn  Ols,a*,i}  =Po)  {j(1)  |^(1),  ®s(i),i}  {j(2)|s(2),  os(2)j}x. .  .xp(0K)  {j{h  ' \s(K\  as(K)  4} 

(3.27) 

which  when  written  in  product  form  is  the  stated  hypothesis.  ■ 


3. 5. 5  Expected  Rewards 

The  expected  rewards  are  necessary  to  explicitly  define  the  objective  function. 
Each  time  the  constellation  is  inspected,  an  action  is  taken  that  has  a  corresponding 
reward.  The  rewards  in  this  context  are  actually  utility  costs  and  are,  therefore, 
negative  values.  Therefore,  the  rewards  will  be  regarded  as  losses.  The  losses  in 
this  model  are  the  sum  of  a  deterministic  utility  cost  to  perform  the  action  chosen 
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and  the  expected  penalty  cost  resulting  from  that  action  causing  the  constellation 
to  transition  into  any  given  state  over  the  next  time  period.  The  penalty  cost  can 
be  assigned  in  a  variety  of  ways.  This  subsection  provides  a  methodology  in  which 
each  constellation  degradation  level  is  assigned  a  fixed  penalty  cost. 

There  is  potentially  a  different  penalty  cost  associated  with  each  constellation 
degradation  level.  Define  Cp(s )  as  the  penalty  cost  associated  with  constellation 
state  s  6  5.  The  total  expected  loss  at  each  decision  epoch  is  the  immediate  cost 
resulting  from  performing  a  maintenance  action  plus  the  expected  penalty  cost  as  a 
result  of  that  action  over  the  next  time  step.  If  we  let  c^k\a)  be  the  cost  to  perform 
maintenance  action  a  for  satellite  k  then  the  reward  at  time  n  when  taking  action  a 
while  in  state  s  is  given  by 

K 

rn(s,a)  =  J^c(fc)(a)  +  '^Cp(sj)pn(j\s,a)  (3.28) 

k= 1  j&S 

where  pn(j\s,a)  is  determined  by  Equation  (3.24). 

Because  the  planning  horizon  is  finite,  there  is  no  decision  to  make  at  the  final 
epoch,  and  the  reward  function  does  not  apply.  Therefore,  the  reward  received  at 
the  final  decision  epoch  is  the  value  of  the  constellation  when  in  state  s  given  by 
rv t(s).  This  is  known  as  the  terminal  reward  and  can  be  defined  by  any  function  of 
the  final  states  of  each  satellite. 

3.6  MDP  Evaluation  Methodology 

Evaluating  a  MDP  requires  that  the  evaluation  criteria  be  established  and  the 
classification  of  the  policy  be  defined.  The  evaluation  criteria  is  determined  by  the 
goals  of  the  decision  maker.  Determining  the  classification  of  the  policy  requires 
knowledge  of  the  evaluation  criteria  and  the  properties  of  the  decision  rules. 
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In  this  research,  the  optimality  criterion  is  the  minimum  expected  total  loss 
(of  utility).  This  optimality  criterion  was  chosen  because  the  goal  of  this  research 
is  to  find  a  policy  that  minimizes  the  total  loss  of  maintaining  a  satellite  over  a 
finite  planning  horizon.  Because  the  total  expected  loss  is  a  a  total  expected  reward 
resulting  in  a  negative  value,  minimizing  the  loss  is  equivalent  to  maximizing  the 
reward. 

A  policy’s  classification  is  based  on  the  classification  of  the  decision  rules  of 
which  it  is  composed.  These  decision  rules  are  classified  by  their  dependence  on  the 
history  of  the  system,  and  the  certainty  with  which  an  action  is  chosen  based  on 
that  decision  rule.  A  decision  rule,  dni  can  be  deterministic  (D)  or  randomized  (R). 
A  deterministic  decision  rule  specifies  the  actions  to  take  for  every  given  state  with 
certainty  while  randomized  decision  rules  specify  a  probability  distribution  on  the 
set  of  actions  for  each  state  [32:21],  A  decision  rule,  dn,  may  also  be  either  history 
dependent  (H)  or  Markovian  (M).  A  history  dependent  decision  rule  determines  the 
actions  or  probability  distribution  on  the  action  set  for  each  history  of  states  and 
actions  up  to  the  current  decision  epoch  while  a  Markovian  decision  rule  determines 
the  actions  or  probability  distribution  on  the  action  set  for  every  given  state  that 
the  system  may  currently  occupy  [32:21],  Combining  both  types  of  decision  rule 
classifications  leads  to  four  possibilities  of  policy  classifications. 

UHR  =  the  set  of  all  policies  that  contain  randomized  history  dependent  deci¬ 
sion  rules. 

IlnD  =  the  set  of  all  policies  that  contain  deterministic  history  dependent 
decision  rules. 

I1MR  =  the  set  of  all  policies  that  contain  Markovian  randomized  decision 
rules. 

ITud  =  the  set  of  all  policies  that  contain  Markovian  deterministic  decision 
rules. 
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Markov  deterministic  policies  (i.e.,  those  belonging  to  IP1iD)  are  the  easiest  to 
implement  and  require  the  least  computing  power.  Also,  by  the  Markov  property 
and  the  total  expected  loss  optimality  criterion,  an  optimal  Markovian  deterministic 
policy  7r*  G  IlMD  is  optimal  for  all  policy  classifications  [32], 

The  single  satellite  MDP  has  a  finite  planning  horizon  with  a  finite  state  and 
action  space.  Also,  the  desired  optimal  policy  of  the  Markov  deterministic  type. 
Therefore,  the  problem  can  be  easily  solved  using  the  well-known  backward  induc¬ 
tion  (value  iteration)  method  for  solving  dynamic  programs.  An  example  of  a  back¬ 
ward  induction  algorithm  is  given  in  Puterman  [32:92],  The  algorithm  computes 
the  optimal  value  of  the  MDP  and  the  optimal  actions  that  produce  the  optimal 
value  for  each  state  in  each  decision  epoch.  As  the  name  implies,  the  algorithm  com¬ 
putes  these  values  for  each  decision  epoch,  beginning  at  N  and  working  backwards 
to  decision  epoch  1  resulting  in  an  optimal  policy,  n*  G  HMD. 

The  backward  recursion  equations  used  in  the  backward  induction  algorithm  to 
determine  the  optimal  value  of  the  MDP  and  the  optimal  actions  at  decision  epoch 
n  for  state  s  are  given  by 


<(s)  =  max  <j  rn(s,  a)  +  ^p„(j|s,  a)v*+1(J)  j>  , 

ieS 


(3.29) 


and 


in)  =  arg  max  <  rn(s,  a)  +  VpnOls,  a)v*n+1(j) 

a  £-  A  „  I  ' 

jes 


(3.30) 


respectively  with  the  boundary  condition 


»Jf(s)  =  r»(s) 


(3.31) 


for  all  s  G  S.  These  equations  are  often  referred  to  as  the  Bellman  functional 
equations  (see  for  example  [18]). 
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The  value  u*(s)  is  the  optimal  value  of  the  MDP  from  decision  epoch  n  forward 
when  in  state  s.  The  value  of  a*{n )  is  the  action  that  produces  the  optimal  value 
of  the  MDP  from  decision  epoch  n  forward.  In  other  words,  u*(s)  is  the  minimum 
expected  loss  incurred  throughout  the  rest  of  the  planning  horizon  as  a  result  of 
choosing  a*(n )  at  decision  epoch  n  and  in  state  s. 

3. 7  Summary 

This  chapter  has  provided  a  methodology  for  obtaining  an  optimal  mainte¬ 
nance  policy  for  a  degrading  satellite  constellation.  First  an  analytical  model  for  the 
stochastic  degradation  process  of  a  satellite  constellation  was  presented.  This  degra¬ 
dation  process  was  modelled  as  a  time-homogeneous  discrete-time  Markov  chain. 
Then  an  optimization  problem  was  formulated  that  can  be  solved  to  find  an  optimal 
maintenance  policy.  The  problem  was  formulated  as  a  stochastic  dynamic  program, 
or  Markov  decision  process  (MDP).  In  Chapter  4,  the  models  developed  in  this 
chapter  are  illustrated  via  two  examples.  The  first  case  is  that  of  a  single  satellite 
constellation,  and  the  second  case  is  a  three  satellite  constellation.  Numerical  results 
are  computed  followed  by  sensitivity  analysis  on  the  model  parameters. 
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4.  Numerical  Results 


In  this  chapter,  we  illustrate  the  model  described  in  Chapter  3  using  notional 
data  in  two  cases.  The  first  case  is  a  one-satellite  constellation,  and  the  second  case 
is  a  three-satellite  constellation.  A  description  of  the  problem  data  is  presented  first 
followed  by  the  explicit  formulations  and  numerical  results  for  each  case.  Finally  we 
provide  a  sensitivity  analysis  on  the  problem  parameters  with  respect  to  the  optimal 
policy  and  optimal  value. 

4-1  Data 

In  order  to  create  a  realistic  notional  data  set,  some  typical  values  for  each 
parameter  were  chosen  by  using  a  number  of  reliable  sources.  The  required  inputs 
are  as  follows: 

•  the  expected  lifetime  of  the  subsystems  that  comprise  each  function, 

•  the  probability  of  successful  repair  for  each  function, 

•  the  cost  to  perform  an  on-orbit  repair  while  in  state  s  denoted  by  Cr(s), 

•  the  probability  that  each  function  will  survive  launch  and  placement  into  orbit, 

•  the  cost  to  replace  a  satellite  denoted  by  Cs,  and 

•  the  penalty  costs  incurred  for  functions  being  non-operational  denoted  by  Cp. 

Expected  lifetime  and  replacement  cost  information  were  obtained  from  the 
Air  Force  Link  Fact  Sheets  [41]  and  Jane’s  Space  Directory  (2002-2003)  [3]  with 
regard  to  the  following  satellite  constellations: 

•  Defense  Support  Program  (DSP), 

•  Defense  Meteorological  Satellite  Program  (DMSP), 

•  Defense  Satellite  Communications  System  (DSCS), 
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•  Navstar  Global  Positioning  System  (GPS),  and 

•  Milstar  Satellite  Communications  System  (Milstar). 

On-orbit  repair  success  rates  and  cost  information  were  gathered  from  Jane’s 
Space  Directory  (2002-2003)  [3]  with  regard  to  the  Hubble  Space  Telescope  (HST) 
servicing  missions  and  launch  success  rates  of  the  Delta  2  rocket  and  space  shuttles. 
Additional  on-orbit  repair  cost  information  was  gathered  from  the  On-Orbit  Servic¬ 
ing  website  [27]  with  regard  to  satellite  repair  robots  that  may  be  available  in  the 
future. 

For  the  purposes  of  this  research,  the  expected  lifetime  of  a  satellite  is  assumed 
to  be  the  design  life.  The  design  life  may  have  a  wide  range  depending  on  the  mission. 
Table  4.1  gives  the  design  life  values  for  some  satellites  in  U.S.  Air  Force  maintained 
constellations. 


Table  4.1  Design  life  of  various  maintained  satellites. 


Constellation 

Design  Life 

DSP 

1.25  years  -  5  years 

DMSP 

1.5  years  -  5  years 

DSCS 

7.5  years  -  10  years 

GPS 

7.5  years  -  10  years 

Milstar 

10  years 

Due  to  high  costs,  on-orbit  repairs  are  extremely  rare;  therefore,  estimates  for 
the  probability  that  a  function  will  be  repaired  successfully  are  rare.  For  this  reason, 
these  probabilities  will  correspond  to  the  probability  of  a  successful  launch  for  a 
space  vehicle,  such  as  a  shuttle  or  a  Delta  rocket,  and  the  successful  repair  rates  of 
the  Hubble  Space  Telescope  (HST).  According  to  Jane’s  Space  Directory  (2002-2003) 
[3:265]  the  Delta  2  rocket  has  a  successful  launch  rate  of  99.01%.  Jane’s  Space  Direc¬ 
tory  (2002-2003)  [3:500]  also  reported  that  as  of  February  2001,  the  shuttle  program 
had  similar  launch  success  rates  with  101/102  or  99.02%  being  successful.  The  HST 
has  had  four  servicing  missions  in  which  each  repair  was  successful,  therefore  it  will 


4-2 


be  assumed  that  the  probability  of  a  successful  repair  given  a  successful  launch  will 
be  high  (  >  0.95). 

According  to  Jane’s  Space  Directory  (2002-2003)  [3],  the  cost  to  perform  an  on- 
orbit  repair  to  the  Hubble  Space  Telescope  in  December  1993  was  $674  million,  $763 
million  in  fiscal  year  2000  dollars  (FY00$).  This  included  $378  million  (FY00$428 
million)  for  the  shuttle  used  to  transport  the  maintenance  crew  to  the  Hubble.  Other 
means  of  performing  satellite  repairs  at  a  much  lower  cost  are  being  developed.  For 
example,  the  On-Orbit  Servicing  website  [27]  reported  that  satellite  repair  robots 
are  being  developed  that  may  potentially  perform  repairs  at  an  estimated  $30  -  $50 
million. 

The  event  that  a  satellite  is  successfully  replaced  is  composed  of  many  subevents. 
The  launch  vehicle  must  successfully  carry  the  satellite  to  the  intended  orbit  and  suc¬ 
cessfully  deploy  the  satellite.  Also,  each  component  on  the  satellite  must  survive  the 
trip  to  orbit  and  successfully  activate  when  on  orbit.  The  probability  of  successful 
launch  has  already  been  established  to  be  close  to  99%.  However,  information  about 
individual  subsystem  survival  rates  on  the  satellite  could  not  be  found.  It  is  assumed 
that  these  probabilities  are  high  (>  0.95). 

For  the  purposes  of  this  research,  the  replacement  cost  is  assumed  to  be  the 
unit  cost  of  a  type  of  satellite.  Table  4.2  gives  the  unit  cost  of  some  typical  satellites 
in  United  States  Air  Force  maintained  constellations  as  cited  in  [41]  and  [3]. 

Table  4.2  Unit  cost  of  some  typical  maintained  satellites. 


Constellation 

Unit  Cost  (millions) 

DSP 

$400 

DMSP 

$88 

DSCS 

$200 

GPS 

$57 

Milstar 

$800 

4-3 


The  penalty  costs  can  be  measured  as  a  monetary  loss  in  some  commercial 
applications.  However,  in  military  applications,  the  penalty  costs  cannot  necessarily 
be  measured  as  a  monetary  loss.  Realistically,  in  the  military  application  a  measure 
of  utility  should  be  assigned  to  relate  the  loss  of  mission  effectiveness  to  the  cost  of 
a  repair  or  replacement.  For  the  purposes  of  this  analysis,  the  penalty  costs  will  be 
assumed  to  be  in  lost  dollars  to  ensure  additivity  of  the  loss  function. 

4  -2  One- Satellite  Example 

In  the  first  example,  consider  a  satellite  that  must  perform  three  functions  to 
be  fully  capable.  We  suppose  a  decision  maker  desires  an  optimal  maintenance  policy 
for  this  satellite  over  the  next  5  years.  The  satellite  will  be  inspected  every  quarter, 
and  a  decision  will  be  made  to  either  do  nothing,  repair,  or  replace  the  satellite.  The 
constellation  contains  one  satellite  (K  =  1)  with  three  functions  (M  =  3);  therefore, 
there  are  eight  possible  degradation  levels  (|<S  =  8).  Since  four  inspections  will  be 
done  every  year  for  five  years  there  are  a  total  of  20  decision  epochs  (TV  =  20). 

4-2.1  One-Satellite  Degradation  Process 

In  the  case  of  a  one-satellite  constellation,  the  state  of  the  constellation  is 
equivalent  to  the  state  of  the  individual  satellite.  Therefore,  the  random  state  of  the 
satellite  at  inspection  epoch  n  is 

X(n)  =  (X1(n),X2(n):X3(n)).  (4.1) 

Because  there  are  three  functions  that  can  each  have  two  possible  states,  the  set  of 
all  possible  states  for  the  satellite  has  cardinality  equal  to  23  =  8.  The  eight  possible 
satellite  states  are  given  in  Table  4.3. 

Because  no  maintenance  actions  are  being  considered  at  this  point,  the  satel¬ 
lite  may  only  transition  into  an  equivalent  state  or  a  more  degraded  state.  Figure 
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Table  4.3  Enumerated  state  space  for  example  1. 


State 

Vector  Representation 

State  Description 

Sl 

(1,1,1) 

fully  capable 

s2 

(1,1,0) 

functions  1  and  2  operational 

S3 

(1,0,1) 

functions  1  and  3  operational 

s4 

(1,0,0) 

function  1  operational 

S5 

(0,1,1) 

functions  2  and  3  operational 

S6 

(0,1,0) 

function  2  operational 

S7 

(0,0,1) 

function  3  operational 

s8 

(0,0,0) 

fully  degraded 

4.1  gives  a  sample  path  that  demonstrates  the  non-increasing  nature  of  the  degrada¬ 
tion  process  when  no  maintenance  is  performed.  This  has  the  effect  of  making  the 
transition  probability  matrix  upper  triangular  as  can  be  seen  in  Equation  (4.2). 


X(B) 

(1,1,1)  - 
(1,1,0)  - 
(1,0,1)  - 
(1,0,0) 
(0,1,1)  - 
(0,1,0)  _ 
(0,0,1)  _ 
(0,0,0)  _ 


0  1 


6 


n 

7 


n 


Figure  4.1  Possible  sample  path  for  {X(n)  :  n  >  0}  when  no  main¬ 
tenance  is  performed  for  example  1. 

The  expected  lifetime  for  this  satellite  is  assumed  to  be  5  years.  It  is  reasonable 
to  assume  that  each  subsystem  on  the  satellite  has  an  expected  lifetime  at  least  as 
long  as  the  entire  satellite.  Therefore,  the  expected  lifetime  of  function  1  is  assumed 
to  be  5.5  years,  the  expected  lifetime  of  function  2  is  assumed  to  be  5.25  years,  and 
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the  expected  lifetime  of  function  3  is  assumed  to  be  6.5  years.  In  other  words,  if 
quarterly  inspections  are  performed  then  the  failure  rates  of  each  function  are  as 
given  in  Table  4.4. 

Table  4.4  Failure  rates  of  each  function  for  example  1. 


Failure  Rate 

Value 

Ai 

0.045455 

A2 

0.047619 

A3 

0.038462 

Using  the  failure  rates  of  Table  4.4  and  the  assumption  of  exponential  lifetimes, 
the  transition  probability  matrix  can  be  easily  determined  to  be 

R1R2R3  R1R2F3  R\  b‘2  R:>}  R\  F2F3  F1R2R3  F1R2F3  F1F2R3  F[  f  2  F3 
0  R1R2  0  /  { 1  /'2  0  FxR2  0  /'1 1'2 

0  0  R1R3  R1F3  0  0  F1R3  F1F3 

0  0  0  R1  0  0  0  Fx 

0  0  0  0  R2R3  R2F3  F2R3  F2F3 

0  0  0  0  0  R2  0  F2 

0  0  0  0  0  0  R3  F3 

0  0  0  0  0  0  0  1 

where  Fm  =  1  —  e_Am  and  Rrn  =  e_Am,  m  =  1,2,3.  Substituting  the  numerical 
values  into  Equation  (4.2)  yields 

0.87675  0.03438  0.04276  0.00168  0.04077  0.00160  0.00199 

0  0.91113  0  0.04444  0  0.04237  0 

0  0  0.91951  0.03606  0  0  0.04276 

0  0  0  0.95556  0  0  0 

P  = 

0  0  0  0  0.91752  0.03598  0.04475 

0  0  0  0  0  0.95350  0 

0  0  0  0  0  0  0.96227 

0  0  0  0  0  0  0 


0.00008 

0.00207 

0.00168 

0.04444 

0.00175 

0.04650 

0.03773 

1 
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The  transition  diagram  is  given  in  Figure  4.2.  Note  that  transitions  may  only 
occur  to  the  same  state  or  a  higher  indexed  (more  degraded)  state  because  no  main¬ 
tenance  is  being  performed. 


Figure  4.2  State  transition  diagram  depicting  all  possible  transitions, 
example  1. 


4-2.2  One-Satellite  MDP  Formulation 

Now  that  the  stochastic  degradation  process  for  the  one-satellite  example  has 
been  described,  we  formulate  the  optimization  problem  as  a  Markov  decision  process 
which  can  be  solved  to  find  an  optimal  maintenance  policy. 

The  inspections  of  the  system  will  occur  quarterly  for  five  years  (N  =  20).  The 
set  of  decision  epochs  is  given  by 

AT  ={1,2,...,  20}.  (4.4) 
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The  state  space  for  the  MDP  model  is  identical  to  the  state  space  described 
in  the  degradation  process.  There  are  three  functions,  each  having  the  ability  to 
either  be  operational,  Xm(n)  =  1,  or  non-operational,  Xm{n)  =  0.  This  produces 
the  random  state  vector 


X(n)  =  (Ad(n),AU(n),X3(n)). 


The  complete  list  of  states  can  be  referenced  in  Table  4.3. 

Suppose  the  decision  maker  feels  it  is  critical  to  have  at  least  one  function 
operational.  In  other  words,  when  all  three  functions  fail,  some  maintenance  action 
must  be  performed.  Furthermore,  when  all  functions  are  operational,  no  on-orbit 
repairs  should  be  performed.  Using  this  information,  an  action  set  can  be  established. 
The  feasible  actions  for  this  scenario  are  listed  in  Table  4.5. 

Table  4.5  Feasible  maintenance  actions  for  example  1. 


Action  Set 

Action 

Definition 

A\ 

®1,1 

do  nothing 

®1,3 

replace 

A-2  ~  A7 

do  nothing 

2 

on  orbit  repair 

&s,  3 

replace 

Ag 

a8,2 

on  orbit  repair 

^8,3 

replace 

The  transition  probabilities  require  knowledge  of  the  probability  of  success¬ 
fully  repairing  function  m,  denoted  by  Hm,  and  the  probability  function  m  survives 
replacement,  denoted  by  Grn.  The  notional  repair  and  replacement  probabilities  for 
each  function  are  summarized  in  Table  4.6. 

Using  the  successful  repair  and  replacement  probabilities  from  Table  4.6  and 
the  transition  probabilities  from  Equation  (4.3),  the  transition  probabilities  for  the 
MDP  can  be  computed.  Choosing  a  maintenance  action  is  equivalent  to  choosing  the 
transition  probability  matrix  by  which  the  satellite  will  evolve  during  the  next  inter¬ 
inspection  period.  For  this  example,  there  are  three  possible  actions  indicating  that 
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Table  4.6  Repair  and  replacement  probabilities  for  each  function  for  example  1. 


Function 

Hm  ( Repair  Probability ) 

Gm  ( Replace  Probability ) 

Function  1 

0.95 

0.975 

Function  2 

0.96 

0.94 

Function  3 

0.97 

0.98 

the  satellite  will  evolve  stochastically  according  to  one  of  three  transition  probability 
matrices  at  each  decision  epoch.  If  action  1  (do  nothing)  is  chosen,  then  the  satellite 
will  evolve  according  to  the  following  transition  probability  matrix  for  the  next  inter¬ 
inspection  interval: 


P  = 


P  = 


R1R2R3 

R1R2F3 

R1F2R3 

R3F2F3 

F1R2R3 

F\  R2F3 

f1f2r3 

F  [  F'2  F3 

0 

R1R2 

0 

RiF2 

0 

F\  R-2 

0 

F1F2 

0 

0 

R1R3 

R1F3 

0 

0 

F1R3 

F1F3 

0 

0 

0 

Ri 

0 

0 

0 

Pi 

0 

0 

0 

0 

R2R3 

R2F3 

F2R3 

F2F3 

0 

0 

0 

0 

0 

R2 

0 

f2 

0 

0 

0 

0 

0 

0 

r3 

f3 

0 

0 

0 

0 

0 

0 

0 

1 

tuting  the  numerical  values  into  (4.5)  yields 

0.87675 

0.03438 

0.04276 

0.00168 

0.04077 

0.00160 

0.00199 

0.00008 

0 

0.91113 

0 

0.04444 

0 

0.04237 

0 

0.00207 

0 

0 

0.91951 

0.03605 

0 

0 

0.04276 

0.00168 

0 

0 

0 

0.95556 

0 

0 

0 

0.04444 

0 

0 

0 

0 

0.91752 

0.03598 

0.04475 

0.00175 

0 

0 

0 

0 

0 

0.95350 

0 

0.04650 

0 

0 

0 

0 

0 

0 

0.96227 

0.03773 

0 

0 

0 

0 

0 

0 

0 

1 

(4.5) 


(4.6) 
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If  action  2  (on-orbit  repair)  is  chosen,  then  the  satellite  will  evolve  according  to  the 
following  transition  probability  matrix  for  the  next  inter-inspection  interval: 


RIR2R3 

R\ R-2F3 

R1F2R3 

R3F2F3 

F1R2R3 

Fi  R/2  F:i 

FIF2R3 

F1F2F3 

R1R2H3 

Ri  R2H3 

R1F2H3 

Ri  F2H3 

F1R2H3 

F-[  R/2  H3 

F\  F2  H 3 

F3F2H3 

R1H2R3 

R3H2F3 

R1H2R3 

R\  H2F3 

F\  H2R3 

F\  H2F3 

F\  II2R3 

F1H2F3 

R3H2H3 

R1H2H3 

R1H2H3 

R1H2H3 

FlH2H3 

F^Hs 

F1H2H3 

F1H2H3 

HIR2R3 

H1R2F3 

Hi  F 2  R3 

HIF2F3 

HIR2R3 

H1R2F3 

H3F2R3 

H1F2F3 

III  R2H3 

H\R2H3 

H1F2H3 

IIIF2U3 

HlR2H3 

h1r2h3 

I/1F2H3 

//  ,  F2H3 

IIIH2R3 

HIH2F3 

//|/72/t3 

H^Fs 

H1H2R3 

H1H2F3 

H1H2R3 

H1H2F3 

III  H2H3 

HiH2H3 

HxH2H3 

Ttii72i73 

H1H2H3 

H1H2H3 

H1H2H3 

H1H2H3 

(4.7) 

Notice  that  the  first  row  of  this  transition  probability  matrix  is  equivalent  to  that  of 
Equation  (4.5).  This  is  because  of  the  assumption  that  only  non-operational  func¬ 
tions  will  be  repaired.  If  all  functions  are  operational,  then  no  functions  will  be 
repaired  and  the  system  will  evolve  according  to  the  underlying  stochastic  degrada¬ 
tion  process.  Substituting  the  numerical  values  into  (4.7)  yields 


0.87675 

0.03438 

0.04276 

0.00168 

0.04077 

0.00160 

0.00199 

0.00008 

0.88379 

0.02733 

0.04310 

0.00133 

0.04110 

0.00127 

0.00200 

0.00006 

0.88273 

0.03461 

0.03678 

0.00144 

0.04105 

0.00161 

0.00171 

0.00007 

0.88982 

0.02752 

0.03708 

0.00115 

0.04138 

0.00128 

0.00172 

0.00005 

0.87164 

0.03418 

0.04251 

0.00167 

0.04588 

0.00180 

0.00224 

0.00009 

0.87865 

0.02717 

0.04285 

0.00133 

0.04624 

0.00143 

0.00226 

0.00007 

0.87759 

0.03441 

0.03657 

0.00143 

0.04619 

0.00181 

0.00192 

0.00008 

0.88464 

0.02736 

0.03686 

0.00114 

0.04656 

0.00144 

0.00194 

0.00006 

(4.8) 
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If  action  3  (replace)  is  chosen,  then  the  satellite  will  evolve  according  to  the  following 
transition  probability  matrix  for  the  next  inter-inspection  interval: 


G!G2G3 

GiG2G3 

GiG2G3 

G!G2G3 

G1G2G3 

Gi  G2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G3 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

G1G2G3 

g,g2g3 

GxG2G3 

GiG2G3 

GIG2G3 

G1G2G3 

GiGaG 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G3 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G3 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

GXG2G3 

GXG2G3 

GiGaGs 

GiG2Ga 

GIG2G3 

G1G2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G3 

GaGaGa 

GiG2G3 

GiG.Ga 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G3 

G1G2G3 

GiG2G3 

GiG2G3 

GiG2G 

Substituting  the  numerical  values  into  (4.9)  yields 


0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

(4.10) 


The  notional  cost  to  perform  on-orbit  repairs  is  assumed  to  be  $450  million 
for  the  space  vehicle  plus  the  function-specific  repair  costs.  The  specific  repair  cost 
of  function  1  is  $35  million,  the  specific  repair  cost  of  function  2  is  $15  million,  and 
the  specific  repair  cost  of  function  3  is  $20  million.  Note  that  economies  of  scale 
exist  for  on  orbit  repairs,  that  is,  it  is  more  costly  to  repair  each  function  at  different 
times  than  it  is  to  repair  multiple  functions  at  the  same  time.  The  cost  of  a  satellite 
replacement  (unit  cost)  is  assumed  to  be  $500  million.  The  notional  penalty  costs, 
assumed  to  be  in  dollars,  are  given  in  Table  4.7. 
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Table  4.7  Penalty  costs,  example  1. 


State 

Penalty  Cost  Cp(s ) 

Sl 

$0 

S2 

$200  million 

S3 

$500  million 

s4 

$600  million 

S5 

$300  million 

s6 

$400  million 

S  7 

$400  million 

s8 

$700  million 

The  expected  rewards  are  obtained  using  Equation  (3.28).  In  the  case  of  K  =  1, 
<S|  =  8  this  equation  simplifies  to 


8 

rn(s,a)  =  r(a )  +  ^Cp(sj)pn(j\s,a).  (4.11) 

3= 1 

The  rewards  for  the  first  IV  —  1  decision  epochs  are  listed  in  Table  4.8. 

Because  these  rewards  look  into  the  next  time  interval  to  compute  the  expected 
rewards,  and  because  there  are  no  decisions  made  at  the  final  decision  epoch,  N  =  20, 
there  is  no  terminal  reward  so  that  r20(si)  =  r 20(^2)  =  •  •  •  =  r2o(ss)  =  0. 


4-2.3  One-Satellite  Optimality  Results 

The  one-satellite  example  was  solved  by  means  of  the  backward  induction 
algorithm  coded  in  MATLAB®.  The  optimal  policy  obtained  using  this  method  is 
presented  in  Table  4.9.  The  policy  given  in  Table  4.9  lists  the  optimal  decision  rule 
for  each  decision  epoch  (Note  that  there  is  no  decision  made  at  the  final  decision 
epoch,  N  =  20).  Each  decision  rule  gives  the  optimal  action  to  take  for  every  possible 
state  the  system  may  be  found  in  at  that  decision  epoch.  For  example,  at  2  years  the 
optimal  decision  rule  is  the  column  of  numbers  below  n  =  8,  (1, 2,  2,  2,  3,  3, 3,  3)T. 
This  decision  rule  is  interpreted  to  mean  that  at  time  n  —  8  the  action  that  produces 
the  minimum  expected  total  loss  over  the  final  three  years  is: 
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Table  4.8  Expected  rewards  given  the  state  and  action  taken  for  the  first  N  —  1 
decision  epochs  as  computed  by  Equation  (3.28),  example  1. 


Action 

State 

Reward 

Value  (millions) 

Do  Nothing 

-Sl 

r n(si,  Gq,i) 

$42.98255 

Do  Nothing 

s2 

T- n(s2,  Ci2,l) 

$227.28197 

Do  Nothing 

3?  ^3,l) 

$499.66476 

Do  Nothing 

S4 

^ 4}  ^4,l) 

$604.44370 

Do  Nothing 

s5 

^5,l) 

$308.77436 

Do  Nothing 

s6 

^"77(^67  ^6,l) 

$413.95091 

Do  Nothing 

s7 

r n(s7,  07^1) 

$411.31939 

Do  Nothing 

s8 

^77(^87  ^8,l) 

N/A 

Repair 

Si 

r n(si,  «1,2) 

N/A 

Repair 

s2 

f n(s2,  a2j2) 

$511.50181 

Repair 

S3 

f n(s3,  a3j2) 

$504.86788 

Repair 

s4 

^7i(^47  ^4,2) 

$523.38278 

Repair 

S5 

^77(^57  ^5,2) 

$529.52992 

Repair 

s6 

f n(s6,  ^6,2) 

$548.05288 

Repair 

S7 

T n(sy,  07j2) 

$541.42931 

Repair 

s8 

r n(s 8,  ^8,2) 

$559.94800 

Replace 

All 

rn(s.,  a. ,3) 

$540.73900 

•  action  1  (do  nothing)  if  the  system  is  found  to  be  in  state  .s  i ; 

•  action  2  (on-orbit  repair)  if  the  system  is  found  to  be  in  states  s2,  s 3,  or  S4; 

•  action  3  (replacement)  if  the  system  is  found  to  be  in  states  S5,  «6,  S7,  or  s$. 

The  total  expected  loss  that  results  from  the  optimal  policy  depends  on  the 
initial  state  of  the  satellite.  Table  4.10  gives  the  minimal  loss  for  each  possible  initial 
state.  For  example,  if  the  system  is  initially  observed  to  be  in  state  S3  then  the 
expected  loss  over  the  5  year  lifetime  of  the  constellation  is  $2298.20  million. 


4-2-4  One-Satellite  Sensitivity  Analysis 

The  optimal  policy  and  resulting  optimal  value  are  the  result  of  a  set  of  fixed 
input  parameters.  These  parameters  are  assumed  to  be  accurate,  but  in  reality  this 
may  not  be  the  case.  For  this  reason  it  is  important  to  know  how  sensitive  the 
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Table  4.9  Optimal  maintenance  policy  for  example  1. 


Decision  Epoch 

(n) 

State 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

-Sl 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

S2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

1 

1 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

1 

s4 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

s5 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

2 

1 

s6 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

1 

■S7 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

1 

s8 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

3 

Table  4.10  Minimum  expected  total  loss  for  every  possible  initial  state  ( millions ) 
for  example  1. 


State 

Minimum  Loss 

State 

Minimum  Loss 

Sl 

$1839.06 

s5 

$2326.33 

s2 

$2304.30 

s6 

$2326.33 

s3 

$2298.20 

s7 

$2326.33 

s4 

$2313.41 

S8 

$2326.33 

optimal  value  and  optimal  policy  are  to  changes  in  the  input  parameters.  In  some 
instances,  it  may  be  more  beneficial  to  implement  a  suboptimal  policy  if  it  is  less 
sensitive  to  changes  in  the  parameters.  For  this  analysis,  we  will  assume  that  the 
system  is  found  to  be  in  the  fully  capable  state  initially,  that  is  X(l)  =  (1, 1, 1). 

Of  particular  interest  are  the  conditions  for  which  the  optimal  policy  includes 
both  on-orbit  repairs  and  satellite  replacements,  henceforth  referred  to  as  mixed 
policies.  As  mentioned  in  the  MDP  formulation,  an  on-orbit  repair  cost  consists  of 
the  space  vehicle  cost  plus  the  function  specific  repair  costs.  For  this  analysis,  the 
function  specific  repair  costs  are  assumed  to  be  fixed  for  each  function.  However, 
the  space  vehicle  cost  is  varied  from  $10  million  to  $1  billion  while  the  replacement 
cost  is  held  constant  to  see  its  effect  on  the  minimum  total  expected  loss.  Likewise, 
the  unit  cost  for  a  satellite  replacement  cost  is  varied  from  $10  million  to  $1  billion 


4-14 


Figure  4.3  One-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  replacement  cost  from  $10  million  to  $1  billion  while  holding 
the  space  vehicle  cost  at  $450  million,  example  1. 


while  the  space  vehicle  cost  is  held  constant  to  see  its  effect  on  the  minimum  total 
expected  loss. 

Figure  4.3  depict  graphically  the  change  in  the  minimum  total  expected  loss 
caused  by  varying  the  unit  cost  of  a  satellite  with  all  else  held  constant.  Figure  4.4 
depicts  graphically  the  change  in  the  minimum  total  expected  loss  caused  by  varying 
the  cost  of  the  space  vehicle  with  all  else  held  constant.  These  two  figures  illustrate 
that  mixed  policies  are  optimal  only  when  the  difference  between  on-orbit  repair  costs 
and  satellite  replacement  costs  is  small.  If  the  on-orbit  repair  cost  is  significantly 
greater  than  the  replacement  cost,  the  optimal  policy  consists  only  of  replacement 
actions.  Likewise,  if  the  replacement  cost  is  significantly  greater  than  the  on-orbit 
repair  cost,  the  optimal  policy  consists  only  of  on-orbit  repair  actions.  Figure  4.4 
illustrates  that  when  the  replacement  cost  is  fixed  at  $500  million,  mixed  policies 
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Figure  4.4  One-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  space  vehicle  cost  from  $10  million  to  $1  billion  while  holding 
the  replacement  cost  at  $500  million,  example  1. 


are  optimal  when  the  space  vehicle  cost  is  approximately  between  $400  million  and 
$500  million,  or  80%  to  100%  of  the  replacement  cost. 

To  gain  further  insight,  another  one-way  analysis  was  done  to  examine  the 
space  vehicle  cost  to  replacement  cost  ratios  for  which  mixed  policies  are  optimal 
when  the  replacement  cost  is  fixed  at  $500  million.  As  can  be  seen  in  Figure  4.5, 
mixed  policies  are  optimal  when  this  ratio  is  between  0.84  and  0.98.  ft  is  worth 
mentioning  that  under  the  conditions  of  this  analysis,  on-orbit  repairs  appear  in  the 
optimal  policy  for  all  ratios  less  than  0.98. 

Next,  we  consider  a  two-way  sensitivity  analysis  to  determine  the  maintenance 
cost  ratios  for  which  mixed  policies  are  optimal.  To  do  this,  the  replacement  cost  is 
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1880  t 


1720  -I - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 

0.80  0.82  0.84  0.86  0.88  0.90  0.92  0.94  0.96  0.98  1.00 

Space  Vehicle  Cost  to  Replacement  Cost  Ratio 

Figure  4.5  One-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  space  vehicle  cost  to  replacement  cost  ratio  from  0.80  to  1.0 
while  holding  the  replacement  cost  at  $500  million,  example  1. 

varied  from  $50  million  to  $1  billion  in  $50  million  increments  while  the  space  vehicle 
cost  is  varied  from  0  to  100%  of  the  replacement  cost. 

As  can  be  seen  in  Figure  4.6,  the  cost  ratios  for  which  mixed  policies  are 
optimal  depends  on  the  magnitude  of  the  maintenance  costs.  The  range  of  cost 
ratios  for  which  mixed  policies  are  optimal  is  wider  for  small  magnitudes  than  for 
large  magnitudes.  Of  notable  interest  is  the  fact  that  when  the  replacement  cost  of 
the  satellite  is  $50  million,  mixed  policies  are  optimal  for  all  cost  ratios  less  than 
0.72.  We  performed  further  analysis  to  determine  that  when  the  replacement  cost  is 
less  than  or  equal  to  $72  million,  there  is  a  cost  ratio  r  such  that  all  policies  for  cost 
ratios  greater  than  r  are  pure  replacement  policies  and  all  policies  for  cost  ratios  less 
than  or  equal  to  r  are  mixed  policies.  This  implies  that  under  the  conditions  of  this 
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Figure  4.6  Two-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  replacement  cost  from  $50  million  to  $1  billion  while  varying 
the  ratio  of  the  space  vehicle  cost  to  replacement  cost  from  0.0  to  1.0, 
example  1. 


example,  satellite  replacements  will  be  included  in  the  optimal  policy  whenever  the 
cost  of  replacement  is  less  than  or  equal  to  $72  million. 


4-3  Three- Satellite  Example 

In  the  second  example,  consider  a  satellite  constellation  consisting  of  three 
satellites.  Satellites  1  and  2  each  have  two  functions  and  satellite  3  has  three  func¬ 
tions.  Also,  suppose  that  a  decision  maker  desires  an  optimal  maintenance  policy  for 
this  satellite  constellation  over  the  next  5  years.  The  constellation  will  be  inspected 
every  quarter,  and  a  decision  will  be  made  to  either  do  nothing,  repair,  or  replace 
each  satellite.  The  constellation  contains  three  satellites  (K  =  3)  with  two  functions 
for  each  of  the  first  two  satellites  ( M ^  =  2,  M =  2)  and  three  functions  on  the 
third  satellite  ( M ®  =  3);  therefore,  there  are  22+2+3  =  128  possible  degradation 


4-18 


levels  (|«5|  =  128)  for  the  constellation.  Since  four  inspections  will  be  done  every 
year  for  five  years  there  are  a  total  of  20  decision  epochs  ( N  =  20). 

4-3.1  Three- Satellite  Degradation  Process 

The  degradation  process  can  be  viewed  as  three  separate  degradation  processes 
because  of  the  assumption  of  mutual  independence  among  satellite  degradation  pro¬ 
cesses.  Each  individual  satellite  has  its  own  state  space  and  degradation  process. 
Two  of  the  satellites  in  this  example  have  two  functions  and  thus  have  four  possi¬ 
ble  degradation  states.  The  remaining  satellite  has  three  functions  giving  it  eight 
possible  degradation  states.  Table  4.11  illustrates  each  satellite’s  individual  state 
space. 


Table  4.11  Individual  satellite  state  spaces,  example  2. 


Satellite  1 

Satellite  2 

Satellite  3 

{  S’  =  (i,D  1 

0(1)  1  4l>  =  (1,0) 

“  41'  =  (o.D 

l  4”  =  (0,0)  , 

f  42)  =  (1,1)  1 
5«J  sf  =  (1,0) 

4’  =  (o.i) 

l  42)  =  (0,0)  , 

5®  =  < 

43'  =  (i,  1. 1)  1 
43)  =  (i.i.o) 
43)  =  (i.o.i) 
43)  =  (1,0,0) 
431  =  (o.i.i) 
431  =  (0,1,0) 
43)  =  (0,0,1) 

,  sf>  =  (0,0,0)  , 

> 

The  overall  state  space  of  the  three  satellite  constellation  model  is  a  three 
component  row  vector  of  each  individual  satellite’s  state  given  by 

X(n)  =  (X(1)(n),X(2)(n),X(3)(n))  (4.12) 
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where  X®(n)  G  S^k\k  =  1,2,3.  Equation  (4.12)  can  also  be  written  as 


(4.13) 

Table  4.12  gives  the  enumerated  state  space  for  the  satellite  constellation.  The  ‘state’ 
columns  give  the  label  for  the  state,  st  where  i  is  the  index  assigned  to  that  state. 
The  ‘vector’  columns  give  the  associated  state  vector  where  each  component  is  the 
index  of  that  individual  satellite’s  state.  For  example,  (1, 1, 1)  means  that  state  si 
of  the  constellation  represents  the  vector  (s^,  s®). 
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Table  4.12  Enumerated  state  space,  example  2. 


State 

Vector 

State 

Vector 

State 

Vector 

State 

Vector 

-Sl 

(1,1,1) 

-S33 

(2,1,1) 

-S65 

(3,1,1) 

S97 

(4,1,1) 

•S  2 

(1.1,2) 

-S34 

(2,1,2) 

-S66 

(3,1,2) 

S98 

(4,1,2) 

(1,1,3) 

-S35 

(2,1,3) 

-S67 

(3,1,3) 

-S99 

(4,1,3) 

S4 

(1,1,4) 

-S36 

(2,1,4) 

-S68 

(3,1,4) 

-SlOO 

(4,1,4) 

s5 

(1,1,5) 

-S37 

(2,1,5) 

-S69 

(3,1,5) 

-SlOl 

(4,1,5) 

s6 

(1,1,6) 

-S38 

(2,1,6) 

-S70 

(3,1,6) 

-S102 

(4,1,6) 

-S7 

(1,1,7) 

-S39 

(2,1,7) 

-S71 

(3,1,7) 

-S103 

(4,1,7) 

s8 

(1,1,8) 

S40 

(2,1,8) 

-S72 

(3,1,8) 

-S104 

(4,1,8) 

s9 

(1,2,1) 

S41 

(2,2,1) 

-S73 

(3,2,1) 

-S105 

(4,2,1) 

SlO 

(1,2,2) 

-S42 

(2,2,2) 

S74 

(3,2,2) 

S106 

(4,2,2) 

•S'l  1 

(1,2,3) 

S43 

(2,2,3) 

S75 

(3,2,3) 

s107 

(4,2,3) 

Sl2 

(1,2,4) 

S44 

(2,2,4) 

S76 

(3,2,4) 

S108 

(4,2,4) 

« 13 

(1,2,5) 

S45 

(2,2,5) 

S77 

(3,2,5) 

-S109 

(4,2,5) 

Sl4 

(1,2,6) 

S46 

(2,2,6) 

-S78 

(3,2,6) 

SllO 

(4,2,6) 

Sl5 

(1,2,7) 

-S47 

(2,2,7) 

S79 

(3,2,7) 

Sill 

(4,2,7) 

Sl6 

(1,2,8) 

S48 

(2,2,8) 

s80 

(3,2,8) 

S112 

(4,2,8) 

s17 

(1,3,1) 

S49 

(2,3,1) 

S81 

(3,3,1) 

S113 

(4,3,1) 

Sl8 

(1,3,2) 

S50 

(2,3,2) 

S82 

(3,3,2) 

Sll4 

(4,3,2) 

Sl9 

(1,3,3) 

S51 

(2,3,3) 

-S83 

(3,3,3) 

S115 

(4,3,3) 

S20 

(1,3,4) 

-S52 

(2,3,4) 

S84 

(3,3,4) 

S116 

(4,3,4) 

S21 

(1,3,5) 

S53 

(2,3,5) 

S85 

(3,3,5) 

S117 

(4,3,5) 

S22 

(1,3,6) 

S54 

(2,3,6) 

S86 

(3,3,6) 

S118 

(4,3,6) 

S23 

(1,3,7) 

S55 

(2,3,7) 

s87 

(3,3,7) 

S119 

(4,3,7) 

S24 

(1,3,8) 

S56 

(2,3,8) 

s88 

(3,3,8) 

S120 

(4,3,8) 

S25 

(1,4,1) 

-S57 

(2,4,1) 

s89 

(3,4,1) 

S121 

(4,4,1) 

-S26 

(1,4,2) 

S58 

(2,4,2) 

S90 

(3,4,2) 

S122 

(4,4,2) 

S27 

(1,4,3) 

S59 

(2,4,3) 

S91 

(3,4,3) 

S123 

(4,4,3) 

S28 

(1,4,4) 

S60 

(2,4,4) 

S92 

(3,4,4) 

Sl24 

(4,4,4) 

S29 

(1,4,5) 

S6i 

(2,4,5) 

-S93 

(3,4,5) 

S125 

(4,4,5) 

S30 

(1,4,6) 

-S62 

(2,4,6) 

S94 

(3,4,6) 

Sl26 

(4,4,6) 

S31 

(1,4,7) 

S63 

(2,4,7) 

S95 

(3,4,7) 

Sl27 

(4,4,7) 

S32 

(1,4,8) 

^64 

(2,4,8) 

s96 

(3,4,8) 

Sl28 

(4,4,8) 
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The  notional  expected  lifetime  for  each  satellite  is  5  years.  It  is  acceptable  to 
assume  that  each  subsystem  on  each  satellite  has  an  expected  lifetime  at  least  as 
long  as  the  satellite  design  life.  In  this  example,  functions  1  and  2  are  assumed  to  be 
identical  on  all  satellites.  The  expected  lifetime  of  function  1  will  be  assumed  to  be 
5.5  years,  the  expected  lifetime  of  function  2  will  be  assumed  to  be  5.25  years,  and 
the  expected  lifetime  of  function  3  will  be  assumed  to  be  6.5  years.  In  other  words, 
if  quarterly  inspections  are  performed,  then  the  failure  rates  of  each  function  are  as 
given  in  Table  4.13. 


Table  4.13  Failure  rates  of  each  function,  example  2. 


Failure  Rate 

Value 

Ai 

0.045455 

a2 

0.047619 

A3 

0.038462 

Using  the  failure  rates  from  Table  4.13  and  the  assumption  of  exponential  life¬ 
times,  the  transition  probability  matrices  for  each  satellite  can  be  easily  determined. 
The  transition  probability  matrix  for  satellites  1  and  2  is  given  by 


R1R2  R[  F‘2  F\  R,‘2  F1F2 

0  /q  0  u 

p  = 

0  0  R2  F2 

0001 

and  the  transition  probability  matrix  for  satellite  3  is  given  by 


(4.14) 


R1R2R3 

R[  R/2F3 

R1F2R3 

R1F2F3 

0 

R\ R2 

0 

R1F2 

0 

0 

R1R3 

R1F3 

0 

0 

0 

Ri 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

FIR2R3 

FIR2F3 

FIF2R3 

F\  F2  F:- 

0 

FlR2 

0 

F\F2 

0 

0 

F1R3 

F\  F:i 

0 

0 

0 

Fi 

R2R3 

R2F3 

F2R3 

F2F3 

0 

R2 

0 

f2 

0 

0 

Ri 

f3 

0 

0 

0 

1 

(4.15) 
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where  Frn  =  1  —  e  Am  and  Rm  =  e  Xm ,  m  —  1,2,  3.  Substituting  the  rate  parameters 
from  Table  4.13  into  (4.14)  and  (4.15)  yields 


P  = 


0.91113  0.04444  0.04237  0.00207 

0  0.95556  0  0.04444 

0  0  0.95350  0.04650 

0  0  0  1 


for  satellites  1  and  2,  and 


(4.16) 


0.87675 

0.03438 

0.04276 

0.00168 

0 

0.91113 

0 

0.04444 

0 

0 

0.91951 

0.03606 

0 

0 

0 

0.95556 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.04077 

0.00160 

0.00199 

0.00008 

0 

0.04237 

0 

0.00207 

0 

0 

0.04276 

0.00168 

0 

0 

0 

0.04444 

0.91752 

0.03598 

0.04475 

0.00175 

0 

0.95350 

0 

0.04650 

0 

0 

0.96227 

0.03773 

0 

0 

0 

1 

for  satellite  3. 


The  overall  constellation  transition  probability  matrix  is  too  large  to  display 
here.  It  contains  28  =  128  rows  and  columns.  In  other  words,  there  are  a  total 
of  128  x  128  =  16,  384  constellation  transition  probabilities.  Because  the  satel¬ 
lites  are  assumed  to  be  independent,  the  transition  probabilities  for  the  entire  con¬ 
stellation  are  the  product  of  the  individual  satellite  transitions.  For  example,  the 
probability  that  the  constellation  transitions  from  state  si  =  (s^,  s^)  to  state 

S33  =  («2  ,  s\  ,  s\  ) is  given  by 


P{«33|si}  =P{41)|S11)}  X  P(S1J)|S12)}  X  P(sl3))lsl3)}-  (4.18) 
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Using  the  individual  satellite  transition  probability  matrices  in  Equations  (4.16)  and 
(4.17),  the  value  of  Equation  (4.18)  is  found  to  be 


p{s33|si}  —  R\F‘i  x  R1R2  X  R1R2R3 

=  0.04444  x  0.91113  x  0.87675 
=  0.03550. 

Some  sample  constellation  transition  probabilities  are  given  in  Table  4.14. 


Table  4.14  Sample  constellation  degradation  transition  probabilities,  example  2. 


Probability 

Transition 

Formula 

Value 

P{s2|si} 

(X)  „(2)  J3K  7  (1)  (2)  (3K 

1*1  >  *1  j  *1  )  •  t*l  >*1  >*2  ) 

R1R2  X  R1R2  X  R1R2F3 

0.02854 

pWki} 

(UP  „(2)  (3)x  7"  (1)  (2)  (3)  \ 

1*1  >  *1  >  *1  )  ’  t*2  i  *2  )  *7  ) 

R  \  F2  x  R,\  F2  x  F\  F2  R.;$ 

3.92650e-6 

p{sA7\sA7} 

(UP  „(2)  (3)x  7  (1)  (2)  (3K 

1*2  >  *2  ) *7  )  1*2  > *2  1 *7  ) 

R\  x  R\  x  R3 

0.87865 

pWkm} 

(JD  U2)  C3)x  7  (1)  (2)  (3K 

1*2  j *2  i*7  )  v*4  ) *2  )* 7  ) 

Fi  x  Ri  x  R3 

0.04086 

4-3.2  Three- Satellite  MDP  Formulation 

The  inspections  of  the  system  will  occur  quarterly  for  five  years.  In  other  words 
N  =  20.  The  set  of  decision  epochs  is  given  by 

AT  ={1,2,...,  20}.  (4.19) 

The  state  space  for  the  MDP  model  is  identical  to  the  state  space  described  in 
the  degradation  process.  There  are  three  satellites,  two  satellites  have  two  functions, 
and  one  satellite  has  three  functions.  Each  function  has  the  ability  to  either  be 
operational,  I® (n)  =  1,  or  non-operational,  Xm\n )  =  0.  This  produces  the  state 
vector 

X(n)  =  (X(1)(n),X(2)(n),X(3}(n))  (4.20) 
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where  X®(n)  G  <S^,  k  =  1,  2,  3.  Equation  (4.20)  can  also  be  written  as 


(4.21) 

The  enumerated  state  space  can  be  referenced  in  Table  4.12. 

There  are  three  satellites,  K  =  3,  and  there  are  three  possible  actions  for 
each  satellite;  do  nothing,  on  orbit  repair,  or  replace.  Therefore,  there  are  33  =  27 
total  combinations  of  actions  at  any  decision  epoch  unless  some  actions  are  deemed 
infeasible  by  the  decision  maker.  As  in  the  single  satellite  example,  when  all  functions 
of  a  satellite  are  operational  no  on-orbit  repairs  may  be  performed  on  that  satellite. 
Also,  the  decision  maker  eliminates  the  possibility  of  doing  nothing  when  all  functions 
of  all  satellites  are  non-operational.  The  enumerated  action  space  is  too  large  to 
display  here.  There  are  128  states  each  having  a  subset  of  27  associated  actions.  The 
upper  bound  for  the  total  number  of  actions  in  the  action  set  is  128  x  27  =  3,456. 
As  an  example,  the  action  space  for  state  sn  is  given  in  Table  4.15.  Recall  that  Sn 
means  that  satellite  1  is  in  state  =  (1, 1),  satellite  2  is  in  state  =  (1,0),  and 
satellite  3  is  in  state  s ^  =  (1,0,1);  therefore,  on-orbit  repairs  are  not  allowed  for 
satellite  1. 

The  transition  probabilities  require  knowledge  of  the  probability  that  function 
m  is  successfully  repaired  (Hm),  and  the  probability  that  function  m  survives  replace¬ 
ment  (Gm).  The  notional  repair  and  replacement  probabilities  for  each  function  are 
summarized  in  Table  4.16. 

Using  the  successful  repair  and  replacement  probabilities  from  Table  4.16  and 
the  transition  probabilities  of  Equations  (4.16)  and  (4.17),  the  transition  probabil¬ 
ities  for  the  MDP  can  be  computed.  Choosing  a  maintenance  action  is  equivalent 
to  choosing  the  transition  probability  matrix  by  which  the  constellation  will  evolve 
during  the  next  inter-inspection  period.  Because  there  is  separate  action  chosen  for 
each  satellite  in  the  constellation  and  because  the  satellites  are  stochastically  inde¬ 
pendent,  each  satellite’s  MDP  transition  probability  matrix  can  be  viewed  separately. 
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Table  4.15  Sample  action  set  for  state  sn,  example  2. 


Action  Set 

Action 

Definition 

An 

<211,1 

do  nothing,  do  nothing,  do  nothing 

Oil, 2 

do  nothing,  do  nothing,  repair 

Qll, 3 

do  nothing,  do  nothing,  replace 

Oil, 4 

do  nothing,  repair,  do  nothing 

Oil, 5 

do  nothing,  repair,  repair 

Oil, 6 

do  nothing,  repair,  replace 

Oil, 7 

do  nothing,  replace,  do  nothing 

Oil, 8 

do  nothing,  replace,  repair 

Qll, 9 

do  nothing,  replace,  replace 

On, 19 

replace,  do  nothing,  do  nothing 

Oil, 20 

replace,  do  nothing,  repair 

Oil, 21 

replace,  do  nothing,  replace 

Oil, 22 

replace,  repair,  do  nothing 

Oil, 23 

replace,  repair,  repair 

Oil, 24 

replace,  repair,  replace 

Oil, 25 

replace,  replace,  do  nothing 

Oil, 26 

replace,  replace,  repair 

Oil, 27 

replace,  replace,  replace 

Table  4.16  Repair  and  replacement  probabilities  for  each  function,  example  2. 


Function 

Hm  ( Repair  Probability) 

Gm  ( Replace  Probability ) 

Function  1 

0.95 

0.975 

Function  2 

0.96 

0.94 

Function  3 

0.97 

0.98 

Because  there  are  three  possible  actions  for  each  satellite,  each  satellite  will  evolve 
stochastically  according  to  one  of  three  transition  probability  matrices  each  decision 
epoch.  If  action  1  (do  nothing)  is  chosen  for  satellites  1  or  2,  the  satellite  will  evolve 
according  to  the  following  transition  probability  matrix  for  the  next  inter-inspection 
interval: 

r  R1R2  R1F2  F1R2  F[  F2 1 


0  1?1  0  Ri 

0  0  R2  F2 

0  0  0  1 


(4.22) 
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If  action  1  (do  nothing)  is  chosen  for  satellite  3,  the  satellite  will  evolve  according 
to  the  following  transition  probability  matrix  for  the  next  inter-inspection  interval: 


RIR2R3 

R[  R2F3 

RIF2R3 

R1F2F3 

FIR2R3 

F-[  R.2  F3 

FIF2R3 

F3F2F3 

0 

R1R2 

0 

R1F2 

0 

FlR2 

0 

F[  F2 

0 

0 

R3R3 

R1F3 

0 

0 

F[  R3 

F\F3 

0 

0 

0 

Ri 

0 

0 

0 

Fi 

0 

0 

0 

0 

R2R3 

R2F3 

F2R3 

F2F3 

0 

0 

0 

0 

0 

R2 

0 

F2 

0 

0 

0 

0 

0 

0 

R‘i 

F3 

0 

0 

0 

0 

0 

0 

0 

1 

(4.23) 

Substituting  the  numerical  values  into  Equations  (4.22)  and  (4.23)  yields 


0.91113 

0.04444 

0.04237 

0.00207 

0 

0.95556 

0 

0.04444 

0 

0 

0.95350 

0.04650 

0 

0 

0 

1 

and 


(4.24) 


0.87675 

0.03438 

0.04276 

0.00168 

0 

0.91113 

0 

0.04444 

0 

0 

0.91951 

0.03605 

0 

0 

0 

0.95556 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0.04077 

0.00160 

0.00199 

0.00008 

0 

0.04237 

0 

0.00207 

0 

0 

0.04276 

0.00168 

0 

0 

0 

0.04444 

0.91752 

0.03598 

0.04475 

0.00175 

0 

0.95350 

0 

0.04650 

0 

0 

0.96227 

0.03773 

0 

0 

0 

1.00000 

If  action  2  (on-orbit  repair)  is  chosen  for  satellites  1  or  2,  the  satellite  will  evolve 
according  to  the  following  transition  probability  matrix  for  the  next  inter-inspection 
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interval: 


N/A 

N/A 

N/A 

N/A 

RiH2 

RiH2 

F\H2 

fxh2 

HJh 

H\F2 

H\R2 

HiF2 

//  i  Hi 

hxh2 

hxh2 

hxh2 

If  action  2  (on-orbit  repair)  is  chosen  for  satellite  3,  the  satellite  will  evolve  according 
to  the  following  transition  probability  matrix  for  the  next  inter-inspection  interval: 


N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

RiR2H3 

R\R2h3 

RiF2H3 

R\  F‘2  H3 

fxr2h3 

F\R2H3 

F\F2H3 

F\F2H3 

R\H2r3 

RiH2f3 

RiH2R3 

RiH2F3 

f±h2r3 

F\H2F3 

fxh2r3 

F\H2F3 

RiH2H3 

RiH2H3 

RiH2H3 

RiH2H3 

fxh2h3 

fxh2h3 

fJi2h3 

fJi2h3 

HxR2R3 

HxR2F3 

HiF2R3 

H\  F2  F3 

H\R2R3 

HiR2F3 

HiF2R3 

H\F2F3 

HxR2H3 

//,  ii>n3 

HxF2H3 

HxF2H3 

HxR2H3 

HxR2H3 

HxF2H3 

HxF2H3 

hxh2r3 

hxh2f3 

hxh2r3 

hxh2f3 

HiH2R3 

HiH2F3 

hxh2r3 

hxh2f3 

hxh2h3 

hxh2h3 

hxh2h3 

hxh2h3 

HiH2H3 

HiH2H3 

HiH2H3 

hxh2h3 

(4.27) 

Substituting  the  numerical  values  into  Equations  (4.26)  and  (4.27)  yields 


N/A 

N/A 

N/A 

N/A 

0.91734 

0.03822 

0.04266 

0.00178 

0.90582 

0.04418 

0.04767 

0.00233 

0.91200 

0.03800 

0.04800 

0.00200 

and 


(4.28) 


N/A 

N/A 

N/A 

N/A 

0.88379 

0.02733 

0.04310 

0.00133 

0.88273 

0.03461 

0.03678 

0.00144 

0.88982 

0.02752 

0.03708 

0.00115 

0.87164 

0.03418 

0.04251 

0.00167 

0.87865 

0.02717 

0.04285 

0.00133 

0.87759 

0.03441 

0.03657 

0.00143 

0.88464 

0.02736 

0.03686 

0.00114 

N/A 

N/A 

N/A 

N/A 

0.04110 

0.00127 

0.00200 

0.00006 

0.04105 

0.00161 

0.00171 

0.00007 

0.04138 

0.00128 

0.00172 

0.00005 

0.04588 

0.00180 

0.00224 

0.00009 

0.04624 

0.00143 

0.00226 

0.00007 

0.04619 

0.00181 

0.00192 

0.00008 

0.04656 

0.00144 

0.00194 

0.00006 

(4.29) 
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If  action  3  (replace)  is  chosen  for  satellites  1  or  2,  the  satellite  will  evolve 
according  to  the  following  transition  probability  matrix  for  the  next  inter-inspection 
interval: 


G1G2 

GiG2 

G\H2 

GiG; 

GiG2 

GiG2 

gxh2 

GlG; 

GiG2 

G\G2 

G\H2 

GiG; 

GiG2 

G\G2 

G[  H2 

GiG; 

If  action  3  (replace)  is  chosen  for  satellite  3,  the  satellite  will  evolve  according  to 
the  following  transition  probability  matrix  for  the  next  inter-inspection  interval: 


G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

G!G2G3 

G!G2G3 

G!G2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

GiG2G3 

g1g2g3 

GiG2G 

G!G2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiG2G3 

GiG2G3 

GiG2G3 

GiG2G 

GiGaGa 

GiG2G3 

GiGaGa 

GiG2G 

(4.31) 


Substituting  the  numerical  values 


into  Equations  (4.30)  and  (4.31)  yields 


0.91650 

0.05850 

0.02350 

0.00150 

0.91650 

0.05850 

0.02350 

0.00150 

0.91650 

0.05850 

0.02350 

0.00150 

0.91650 

0.05850 

0.02350 

0.00150 

and 


0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.89817 

0.01833 

0.05733 

0.00117 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

0.02303 

0.00047 

0.00147 

0.00003 

(4.33) 
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Knowledge  of  each  satellite’s  individual  transition  probabilities  is  sufficient  to 
compute  the  overall  constellation’s  transition  probabilities.  The  overall  transition 
probabilities  are  computed  using  Equation  (3.24)  with  K  =  3: 

3 

Pn{j\s,as,i}  =  n^0  {j(fc)|s(fc),as«,J  ,  n  =  1,2,. .  .,7V-  1.  (4.34) 

k= 1 


The  constellation  transition  probability  matrices  are  too  large  to  display  here.  There 
are  a  total  of  27  actions  and  128  states  for  the  constellation.  Therefore,  each  action 
produces  a  transition  probability  matrix  that  has  128  rows  and  columns.  This  means 
that  there  are  a  total  of  128  x  27  x  128  =  442,  368  transition  probabilities  for  the  three 
satellite  constellation  MDP  example.  Table  4.14  can  be  referenced  as  an  example  of 
transition  probabilities  when  the  action  is  ‘do  nothing’  for  all  satellites. 

The  cost  to  perform  on-orbit  repairs  for  the  multiple  satellite  model  is  com¬ 
posed  of  the  cost  of  the  space  vehicle,  and  the  function  specific  maintenance  costs. 
The  cost  to  perform  a  replacement  is  composed  of  the  the  cost  of  the  unit  cost  of 
the  replacement  satellite  plus  any  function  upgrades.  These  costs  are  summarized 
in  Table  4.17. 


Table  4.17  Notional  on-orbit  repair  and  replacement  costs  (millions),  example  2. 


Cost 

Satellite  1 

Satellite  2 

Satellite  3 

Space  Vehicle 

$450 

$450 

$450 

Function  1  Repair 

$35 

$35 

$35 

Function  2  Repair 

$15 

$15 

$15 

Function  3  Repair 

N/A 

N/A 

$20 

Unit  Cost 

$470 

$470 

$470 

Function  Upgrade  Cost 

$30 

$30 

$50 

Suppose  that  for  the  satellite  constellation  to  be  considered  fully  capable,  two 
satellites  must  have  function  1  operational,  and  two  satellites  must  have  function 
2  operational.  In  other  words,  function  1  must  be  non-operational  on  two  or  more 
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satellites  before  a  penalty  cost  is  assessed.  The  same  can  be  said  about  function  2. 
Also,  suppose  that  function  3  improves  performance  of  the  constellation,  however, 
the  mission  can  be  completed  successfully  when  the  function  is  non-operational. 
In  other  words,  a  small  penalty  cost  is  assessed  when  function  3  fails.  Table  4.18 
summarizes  the  notional  penalty  costs  for  this  scenario. 


Table  4.18  Notional  penalty  costs  assigned  (millions),  example  2. 


Number  Operational 

Function  1 

Function  2 

Function  3 

3 

$0 

$0 

N/A 

2 

$0 

$0 

N/A 

1 

$500 

$450 

$0 

0 

$650 

$700 

$200 

The  rewards  for  the  three  satellite  model  are  found  using  Equation  (3.28)  with 
K  =  3  and  |<S|  =  128.  That  is 


3  128 

rn(s,a )  =  J^r(k)(a)  +  ^Cp(j)pn(j\s,a) 
k= 1  j=l 

where  pn(j\s,a)  is  found  with  Equation  (4.34).  The  enumerated  list  of  rewards  is 
too  large  to  display  here,  however,  Table  4.19  gives  some  example  rewards. 

Table  4.19  Expected  rewards  for  the  first  N  —  1  decision  epochs,  example  2. 


State 

Action 

Reward 

Expected  Value  Formula 

Value  (millions) 

Sl 

(1,1,1) 

^l,l) 

0  +  'EiesCpU)p{j\shahi} 

$13.2877 

Sl 

(1,1,3) 

f- n  (Sl,  Ol,3) 

$520  +  ; 52ieSCpU)p{j\si,ai,3 } 

$529.4563 

Sl28 

(2,1.2) 

T n(>Sl28,  0128, ll) 

$500  +  $520  +  E,e5  Cp(j)p{j\si28  ,  0128, 11} 

$1110.805 

Sl28 

(1,3,1) 

T n(>Sl28,  Ol28,7) 

$500  +  )C?es  CpU)p{j\si28,  0128,7} 

$1668.75 

Because  these  rewards  look  into  the  next  time  interval  to  compute  the  expected 
rewards,  and  because  there  are  no  decisions  made  at  the  final  decision  epoch,  N  =  20, 
there  is  no  terminal  reward  so  that  r20(si)  =  r20(s2)  =  •  •  •  =  r20(si28)  =  0. 
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4-3.3  Three- Satellite  Optimality  Results 

The  three-satellite  example  was  solved  by  means  of  the  backward  induction 
algorithm  coded  in  MATLAB®.  The  optimal  policy  found  using  this  method  is 
presented  in  Tables  4.20,  4.21,  and  4.22.  This  policy  lists  the  optimal  decision  rule 
for  each  decision  epoch.  Each  decision  rule  gives  the  optimal  action  to  take  for  every 
possible  state  the  system  may  be  found  in  at  that  decision  epoch.  For  example,  if  the 
constellation  is  found  to  be  in  state  s2 6  (representing  the  state  vector  (1,4,2))  after 
2  years  (n  =  8)  then  the  optimal  action  to  take  is  action  8  (representing  the  action 
vector  (1,3,2)).  In  other  words,  suppose  that  at  inspection  epoch  8  the  following 
observation  of  the  constellation  is  made: 

•  Satellite  1  is  fully  capable, 

•  satellite  2  is  fully  non-capable,  and 

•  satellite  3  has  functions  1  and  2  operational  and  function  3  non-operational. 

Then  the  actions  for  each  satellite  that  will  produce  the  minimum  total  expected 
loss  over  the  rest  of  the  constellation’s  lifetime  are  as  follows: 

•  Do  nothing  for  satellite  1; 

•  replace  satellite  2;  and 

•  perform  an  on-orbit  repair  to  satellite  3. 

The  total  expected  loss  that  results  from  the  optimal  policy  depends  on  the 
initial  state  of  the  constellation.  Table  4.23  gives  the  optimal  loss  for  every  possible 
initial  state. 
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Tabic  4.21  Three-satellite  optimal  policy  (states  S43  —  S84),  example  2. 
|  Decision  Epoch  (n) 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

13 

11 

11 

5 

5 

5 

5 

5 

5 

5 

5 

11 

5 

5 

10 

10 

10 

10 

4 

4 

10 

4 

10 

4 

4 

4 

10 

11 

11 

11 

11 

5 

5 

5 

5 

11 

5 

5 

5 

5 

11 

11 

11 

5 

5 

5 

5 

5 

11 

11 

5 

5 

5 

12 

12 

11 

11 

5 

5 

5 

11 

5 

11 

5 

5 

5 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

13  13 

5  5 

7  7 

5  5 

5  5 

6  6 

1  1 
2  2 
1  1 
2  2 
19  19 

3  3 


1  1 
2  2 


10  10 
3  3 


7  7 


4  10 


13  13  13 


7  7  7 


1  1  1 

2  2  2 

1  1  1 

2  2  2 

19  19  19 

3  3  3 

3  3  3 

3  3  3 

1  1  1 

2  2  2 


10  10  10 

3  3  3 

2  2  2 

3  3  3 

7  19  7 

8  8  20 

10  4  4 


13  13 

5  5 


1  1 
2  2 
1  1 
2  2 
19  19 

3  3 


1  1 

2  2 


10  10 
3  3 


1  1  1 

2  2  2 

1  1  1 

2  2  2 

19  19  19 

3  3  3 

3  3  3 

3  3  3 

1  1  1 

2  2  2 


10  10  10 

3  3  3 

2  2  2 

3  3  3 


7  7  7  7 


14  |  15  |  16 
13  I  13  I  13 
5  5  5 

4  4  4 

5  5  5 

5  5  11 

5  5  5 

1  1  1 

2  2  2 

10  10  10 

2  2  2 

4  4  4 

3  3  3 

2  2  2 

3  3  3 

4  4  4 

5  5  5 

13  13  13 

5  5  5 

7  7  7 

5  5  5 

5  5  5 

5  5  5 

1  1  1 

2  2  2 

1  1  1 

2  2  2 

19  19  19 

3  3  3 

3  3  3 

3  3  3 

1  1  1 

2  2  2 


1  1 

2  2 


13  13  13  13  13  13 


7  7 


1  1  1 

2  2  2 

1  1  1 

2  2  2 

19  19  19 

3  3  3 

3  3  3 

3  3  3 

1  1  1 

2  2  2 


17  |  18  |  19 
13  I  13  I  1 
5  5  1 

4  4  1 

5  4  1 

5  5  1 

5  5  1 

1  1  1 

2  1  1 

10  2  1 

2  2  2 


10  10  10  10  10  10 

3  3  3  3  3  3 

2  2  2  2  2  2 

3  3  3  3  3  3 

19  7  7  7  7  7 


13  13  1 
5  5  1 


5  5 

7  7 


7  7 

7  7 


1  1  1 

2  1  1 

1  1  1 

2  1  1 

19  19  1 

3  3  2 

3  3  1 

3  3  3 


1  1 
2  1 
4  2 

2  2 


1  1 
2  1 


4  4 


10  4 


4  4  4 
8  20  8 


4  4  4 


2  2  2 

10  10  1 

3  3  2 

2  2  2 

3  3  3 

7  7  1 

8  19  1 

4  4  1 

8  4  1 


Table  4.22  Three-satellite  optimal  policy  (states  S85  —  si2s);  example  2. 
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Table  4.23  Three-satellite  minimum  expected  total  loss  for  every  possible  initial  state 
{millions). 


State 

Loss 

State 

Loss 

State 

Loss 

State 

Loss 

Sl 

2130.4974 

S33 

2566.41431 

s65 

2598.18333 

S97 

2627.26077 

S'2 

2595.59829 

S34 

3031.51996 

S66 

3063.28906 

598 

3092.36163 

«3 

2490.08459 

S35 

2949.36661 

S67 

2947.86501 

599 

2987.04673 

s4 

2608.12886 

S36 

3041.30435 

S68 

3075.88774 

SlOO 

3104.85193 

s5 

2521.32085 

s  37 

2945.95808 

S69 

3008.96892 

SlOl 

3008.96892 

s6 

2632.8883 

S  38 

3068.74375 

S70 

3095.01841 

S102 

3129.59855 

S7 

2630.329 

S39 

3063.43577 

S71 

3095.01841 

Sl03 

3126.99905 

s8 

2636.42943 

S40 

3078.29776 

S72 

3095.01841 

Sl04 

3133.46205 

S9 

2566.41431 

S41 

3025.68798 

S73 

3019.62094 

5105 

3063.34134 

s  1 0 

3031.51996 

S42 

3490.79364 

S74 

3484.74032 

5106 

3528.44708 

Sn 

2949.36661 

S43 

3408.64026 

S75 

3407.16404 

5107 

3446.32048 

Sl2 

3041.30435 

S44 

3500.5696 

s76 

3494.51039 

Sl08 

3538.22295 

«13 

2945.95808 

S45 

3405.24506 

S77 

3436.24471 

5109 

3439.46252 

S 14 

3068.74375 

S46 

3528.0175 

s78 

3522.31108 

SllO 

3565.68637 

Sl5 

3063.43577 

S47 

3522.7011 

s  79 

3519.27779 

Sill 

3560.36981 

Sl6 

3078.29776 

S48 

3537.58864 

S80 

3522.31108 

5112 

3575.18845 

Sl7 

2598.18333 

S49 

3019.62094 

S81 

3085.77006 

5113 

3085.77006 

Sl8 

3063.28906 

S50 

3484.74032 

S82 

3550.87597 

Sll4 

3550.87597 

Sl9 

2947.86501 

S51 

3407.16404 

S83 

3438.15927 

S115 

3441.31771 

S20 

3075.88774 

S52 

3494.51039 

^84 

3563.43461 

Sll6 

3563.43461 

S21 

3008.96892 

S53 

3436.24471 

s85 

3496.56847 

Sll7 

3496.56847 

S22 

3095.01841 

S54 

3522.31108 

S86 

3582.5561 

Sll8 

3582.5561 

S23 

3095.01841 

S55 

3519.27779 

587 

3582.5561 

S119 

3582.5561 

S24 

3095.01841 

S56 

3522.31108 

5  88 

3582.5561 

5120 

3582.5561 

S25 

2627.26077 

s  57 

3063.34134 

589 

3085.77006 

5121 

3124.29585 

S26 

3092.36163 

s58 

3528.44708 

590 

3550.87597 

5122 

3589.39668 

S  27 

2987.04673 

s  59 

3446.32048 

5gi 

3441.31771 

5123 

3484.01603 

S28 

3104.85193 

s60 

3538.22295 

592 

3563.43461 

Sl24 

3601.84751 

S29 

3008.96892 

S61 

3439.46252 

593 

3496.56847 

5 125 

3496.56847 

S30 

3129.59855 

S62 

3565.68637 

594 

3582.5561 

5126 

3626.57863 

S31 

3126.99905 

S63 

3560.36981 

595 

3582.5561 

5127 

3623.93977 

S32 

3133.46205 

S64 

3575.18845 

596 

3582.5561 

5128 

3630.77118 
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4-3-4  Three- Satellite  Sensitivity  Analysis 

As  in  the  one-satellite  example,  the  optimal  policy  and  resulting  optimal  values 
are  the  result  of  a  set  of  fixed  input  parameters  that  may  or  may  not  be  accurate. 
Therefore,  the  conditions  for  which  mixed  policies  are  optimal  are  also  considered 
for  the  three-satellite  model.  As  mentioned  in  the  MDP  formulation,  an  on-orbit 
repair  cost  consists  of  the  space  vehicle  and  the  function  specific  repair  cost.  For 
this  analysis,  the  function  specific  repair  costs  for  each  satellite  are  assumed  to  be 
constant.  However,  the  space  vehicle  cost  is  varied  from  $10  million  to  $1  billion 
to  see  its  effect  on  the  minimum  total  expected  loss.  Likewise,  the  unit  cost  for  a 
replacement  is  varied  from  $10  million  to  $1  billion  to  see  its  effect  on  the  minimum 
total  expected  loss.  Due  to  the  large  number  of  states,  only  state  si,  fully  capable, 
is  considered  for  the  starting  state. 


Space  Vehicle  Cost($M) 


Figure  4.7  One-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  space  vehicle  cost  from  $10  million  to  $1  billion  while  holding 
the  unit  cost  at  $470  million,  example  2. 
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Figure  4.8  One-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  unit  cost  from  $10  million  to  $1  billion  while  holding  the 
space  vehicle  cost  at  $450  million,  example  2. 

Figure  4.7  depicts  graphically  the  change  in  the  minimum  total  expected  loss 
caused  by  varying  the  cost  of  the  space  vehicle  with  all  else  held  constant.  Figure  4.8 
depicts  graphically  the  change  in  the  minimum  total  expected  loss  caused  by  varying 
the  unit  cost  of  a  satellite  with  all  else  held  constant.  As  in  the  one-satellite  model, 
the  two  figures  illustrate  that  mixed  policies  are  optimal  only  when  the  difference 
between  on-orbit  repair  costs  and  satellite  replacement  costs  is  small.  If  the  on-orbit 
repair  cost  is  significantly  larger  than  the  replacement  cost,  then  the  optimal  policy 
consists  only  of  replacement  actions.  Likewise,  if  the  replacement  cost  is  significantly 
larger  than  the  on-orbit  repair  cost,  then  the  optimal  policy  consists  only  of  on-orbit 
repair  actions.  Looking  at  Figure  4.7  we  can  see  that  when  the  unit  cost  of  a 
replacement  satellite  is  fixed  at  $470  million,  then  mixed  policies  are  optimal  when 
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the  space  vehicle  cost  is  between  $420  million  and  $510  million,  or  approximately 
0.9  to  1.09  of  the  unit  cost  of  a  replacement  satellite. 


Figure  4.9  One-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  space  vehicle  cost  to  unit  replacement  cost  ratio  from  0.85 
to  1.15  while  holding  the  unit  replacement  cost  at  $470  million,  example 
2. 

For  further  insight,  another  one-way  analysis  was  done  to  examine  the  space 
vehicle  cost  to  replacement  cost  ratios  for  which  mixed  policies  are  optimal  when  the 
unit  replacement  cost  is  fixed  at  $470  million.  As  seen  in  Figure  4.9,  mixed  policies 
are  optimal  when  the  ratio  is  between  0.91  and  1.08.  It  is  worth  mentioning  that 
under  the  assumptions  of  this  analysis,  on-orbit  repairs  appear  in  the  optimal  policy 
for  all  ratios  less  than  1.08. 

Next,  we  consider  a  two-way  sensitivity  analysis  to  determine  the  maintenance 
cost  ratios  for  which  mixed  policies  are  optimal.  To  do  this,  the  unit  replacement 
cost  is  varied  from  $50  million  to  $1  billion  in  $50  million  increments  while  the  space 
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vehicle  cost  is  varied  from  0  to  120%  of  the  unit  replacement  cost.  The  results  of 
this  analysis  are  depicted  graphically  in  Figure  4.10. 


Figure  4.10  Two-way  sensitivity  analysis  of  the  minimum  total  expected  loss  when 
varying  the  unit  replacement  cost  from  $50  million  to  $1  billion  while 
varying  the  ratio  of  the  space  vehicle  cost  to  unit  replacement  cost  from 
0.0  to  1.2,  example  2. 


As  can  be  seen  in  Figure  4.10,  the  cost  ratios  for  which  mixed  policies  are 
optimal  depends  on  the  magnitude  of  the  maintenance  costs.  The  range  of  cost 
ratios  for  which  mixed  policies  are  optimal  is  wider  for  small  magnitudes  than  for 
large  magnitudes.  Of  notable  interest  is  the  fact  that,  under  the  conditions  of  this 
analysis,  when  the  unit  cost  of  a  replacement  satellite  is  less  than  $200  million,  the 
maintenance  cost  ratios  for  which  a  pure  replacement  policy  is  optimal  are  greater 
than  1.20.  Therefore,  we  performed  further  analysis  to  determine  the  ratios  for  which 
pure  replacement  policies  would  be  optimal  when  the  unit  cost  of  a  replacement 
satellite  takes  on  the  values  of  $150  million,  $100  million,  and  $50  million.  The 
results  of  this  analysis  are  displayed  in  Table  4.24. 
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Table  4.24  Ratios  for  which  pure  replacement  policies  are  optimal  when  the  unit  cost 
of  a  satellite  replacement  is  less  thatn  $200  million,  example  2. 


Unit  Replacement  Cost 

Ratio 

$150  million 

1.22 

$100  million 

1.34 

$50  million 

1.67 

4-4  Summary 


In  this  chapter  the  model  described  in  Chapter  3  was  illustrated  in  two  numer¬ 
ical  examples.  The  parameter  values  of  each  example  were  of  a  notional  nature,  but 
provided  realistic  scenarios.  In  each  example,  the  stochastic  degradation  process  was 
described  first  followed  by  an  optimization  problem  formulation  in  the  framework  of  a 
Markov  decision  process.  The  solution  to  the  MDP  was  an  easily  interpretable  main¬ 
tenance  policy  which  yields  the  minimum  total  expected  loss  over  a  finite  planning 
horizon.  A  sensitivity  analysis  of  the  model  parameters  determined  the  conditions 
for  which  maintenance  policies  including  on-orbit  repairs  would  be  optimal. 

Chapter  5  concludes  this  thesis  with  final  remarks  in  three  areas.  First,  a 
summary  of  this  research  is  presented  with  an  emphasis  on  the  specific  contributions. 
Next,  a  description  of  the  insights  gained  throughout  this  research  is  presented. 
Finally,  suggestions  are  made  as  to  the  most  fruitful  areas  of  future  research. 
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5.  Conclusions  and  Future  Research 

Satellite  constellations  are  extremely  valuable  assets  in  both  the  public  and 
private  sectors.  Millions  of  customers  all  over  the  world  rely  on  the  services  provided 
by  constellations  every  day.  However,  satellites  in  a  constellation  may  degrade  over 
time  resulting  in  a  reduction  in  the  quality  of  service  provided.  Therefore,  it  is  desir¬ 
able  to  determine  a  strategy  to  minimize  the  losses  incurred  when  the  constellation 
operates  in  a  degraded  state. 

Currently,  satellite  constellation  budget  planners  view  satellite  constellation 
maintenance  as  a  resource  allocation  problem.  Any  analysis  done  to  aide  in  the 
maintenance  planning  process  is  focused  on  determining  which  type  of  satellite  ac¬ 
quisition  would  be  most  beneficial  without  exceeding  a  fixed  budget.  This  method 
may  result  in  extremely  high  (and  unnecessary)  expenditures.  A  better  approach 
may  be  to  determine  a  maintenance  policy  that  balances  the  losses  incurred  from 
service  degradation  with  the  costs  of  maintaining  a  satellite  constellation.  This  the¬ 
sis  addressed  the  satellite  constellation  maintenance  problem  within  the  framework 
of  an  optimal  maintenance  policy  problem. 

Optimal  maintenance  policies  are  an  important  area  of  study  in  operations 
research.  Most  research  in  this  area  is  concerned  with  hireling  a  policy  of  preventive 
maintenance  which  minimizes  the  long-run  cost  per  unit  time.  The  majority  of  these 
models  are  for  single-unit  systems;  however,  interest  in  the  multi-unit  model  has  been 
steadily  increasing  over  the  past  two  decades.  Multi-unit  models  most  often  consider 
systems  with  increasing  failure  rates  or  systems  with  identical,  constant  failure  rates. 
This  thesis  has  extended  the  literature  by  presenting  a  methodology  to  determine  an 
optimal  maintenance  policy  for  a  multi-unit  system  in  which  each  unit  may  operate  in 
a  partially  degraded  state.  A  satellite  constellation  is  a  multi-unit  system  composed 
of  multi-component  satellites.  The  satellite  components  are  the  functions  defined  in 
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Chapter  3  and  are  assumed  to  possess  a  constant  failure  rate;  however,  the  functions 
are  not  necessarily  identical. 

In  order  to  develop  an  optimal  maintenance  policy,  we  first  studied  the  stochas¬ 
tic  evolution  of  an  unmaintained  satellite  constellation.  Chapter  3  showed  that  under 
a  minimal  set  of  assumptions,  the  stochastic  degradation  of  a  satellite  constellation 
may  be  modelled  as  a  discrete-time  Markov  chain.  The  stochastic  degradation  model 
was  then  used  to  develop  an  optimization  problem  through  which  an  optimal  reactive 
maintenance  policy  could  be  determined.  This  optimization  problem  was  formulated 
as  a  finite-horizon  Markov  decision  process. 

After  the  analytical  model  had  been  developed,  two  numerical  examples  were 
presented  to  illustrate  its  utility.  The  first  example  was  a  one-satellite  constellation 
with  eight  possible  capability  levels  and  three  possible  maintenance  actions.  The 
second  example  was  a  three-satellite  constellation  with  128  possible  capability  lev¬ 
els  and  27  possible  maintenance  actions.  Numerical  results  were  computed  for  each 
example  using  the  backward  induction  algorithm  of  Puterman  [32:92],  Furthermore, 
a  sensitivity  analysis  was  performed  on  the  parameters  of  each  example  to  deter¬ 
mine  the  conditions  under  which  on-orbit  repairs  would  be  a  part  of  the  optimal 
maintenance  policy. 

This  research  provides  further  insight  into  the  satellite  constellation  mainte¬ 
nance  problem  and  should  serve  as  a  building  block  for  future  research.  As  in  any 
mathematical  model,  simplifying  assumptions  were  made  in  order  to  ensure  tractabil- 
ity  of  an  analytical  solution.  Future  research  may  include  the  relaxing  of  each  of  these 
assumptions  to  increase  the  applicability  of  the  model  to  more  complex  real  world 
problems.  A  discussion  of  these  assumption  relaxations  follows. 

This  endeavor  considers  satellites  with  functions  whose  lifetime  distributions 
are  assumed  to  be  mutually  independent.  However,  some  functions  may  actually 
become  non-operational  as  a  result  of  some  other  function  becoming  non-operational. 
Similarly,  a  redundant  function  may  be  switched  to  operational  as  a  result  of  another 
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function  becoming  non-operational.  These  dependencies  could  be  incorporated  into 
the  current  model  to  more  accurately  depict  the  degradation  of  various  satellites. 
Incorporating  these  dependencies  could  complicate  the  reward  structure  and  would 
require  knowledge  of  the  correlation  structure  of  the  dependent  variables  but  would 
not  increase  the  cardinality  of  the  state  or  action  space.  An  example  of  how  this 
relaxation  could  be  implemented  follows. 

Consider  two  functions  of  a  surveillance  satellite  -  inertial  stabilization  (X\) 
and  photographic  imaging  (X2).  If  the  inertial  stabilization  function  of  a  surveillance 
satellite  becomes  non-operational,  then  the  optical  lenses  may  become  oriented  in  the 
wrong  direction  thereby  making  the  photographic  imaging  function  non-operational. 
This  dependency  could  be  modelled  by  defining  the  event  Xx  =  0  as  being  equivalent 
to  the  event  (Xi,X2)  =  (0,0).  In  other  words,  if  the  inertial  stabilization  function  is 
non-operational,  the  photographic  imaging  function  is  also  non-operational.  Incor¬ 
porating  dependencies  with  a  positive  correlation  structure  such  as  this  will  result 
in  decreasing  the  cardinality  of  the  state  space. 

We  also  assumed  that  the  degradation  of  satellites  within  a  constellation  are 
independent.  This  implies  that  the  operational  status  of  the  functions  of  a  satellite 
do  not  affect  the  operational  status  of  any  functions  on  any  other  satellites.  This 
assumption  may  not  be  accurate  for  some  constellations  such  as  those  with  satellites 
that  act  as  relays.  In  this  case,  a  non-operational  relay  channel  may  cause  a  commu¬ 
nication  channel  on  another  satellite  to  become  non-operational.  These  dependencies 
could  be  modelled  in  the  same  manner  as  described  for  dependent  functions  on  the 
same  satellite. 

We  assumed  that  if  two  or  more  satellites  have  the  same  function,  then  the 
failure  rate  of  the  function  is  identical  for  both  satellites.  In  reality,  if  two  satellites 
have  the  same  function,  the  newer  satellite  may  have  a  lower  failure  rate  due  to 
technological  improvements  that  have  increased  the  function’s  reliability.  In  such 
a  case,  this  assumption  can  be  relaxed  by  naming  the  function  differently  for  each 
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satellite  and  assigning  each  a  distinct  failure  rate.  This  would  allow  the  model  to 
be  applied  to  more  complex  constellations  and  will  have  no  impact  on  the  size  or 
computational  intensity  of  the  problem. 

There  is  a  large  setup  cost  associated  with  performing  an  on-orbit  repair  due 
to  the  inaccessible  nature  of  the  space  environment.  Therefore,  we  assumed  that  all 
non-operational  functions  on  a  satellite  would  be  repaired  at  the  same  time  when  the 
action  on-orbit  repair  is  chosen.  As  space  maintenance  technology  evolves,  the  setup 
cost  may  decrease  making  it  economically  beneficial  to  consider  performing  on-orbit 
repairs  to  non-operational  functions  at  different  times.  It  may  also  be  beneficial  to 
consider  performing  on-orbit  preventive  maintenance  to  operational  functions  if  the 
functions  exhibit  an  increasing  failure  rate.  Incorporating  either  of  these  assumption 
relaxations  into  the  model  would  have  the  effect  of  increasing  the  dimensionality  of 
the  action  space. 

We  assumed  that  if  a  function  is  successfully  repaired  or  survives  a  replacement 
attempt  then  the  function  will  remain  operational  until  the  next  inspection  epoch. 
This  implies  that  the  probability  of  an  infant  mortality  is  included  in  the  probability 
of  a  successful  maintenance  attempt.  An  interesting  extension  would  be  to  model  the 
the  period  during  which  the  function  may  be  subject  to  an  infant  mortality  separate 
from  the  probability  of  successful  maintenance.  Implementing  this  extension  could 
make  the  model  more  accurately  represent  the  real  world  system. 

We  assumed  that  a  successful  repair  would  return  the  function  to  new  condi¬ 
tion.  That  is,  the  function  would  become  operational  and  would  retain  a  new  condi¬ 
tion  failure  rate.  An  on-orbit  repair  may  actually  result  in  returning  the  function  to 
the  operational  state,  but  with  an  increased  failure  rate  due  to  stresses  incurred  at 
failure.  An  interesting  extension  would  be  to  consider  failure  rates  to  be  dependent 
on  the  number  of  repairs  performed  on  the  function.  One  method  of  accomplishing 
this  could  be  to  model  each  function’s  operational  status  as  a  bivariate  process  giving 
the  status  and  number  of  failures  of  the  function.  This  would  greatly  increase  the 
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cardinality  of  the  state  space  making  the  problem  more  computationally  intensive; 
however,  incorporating  this  extension  would  allow  more  complex  constellations  to  be 
modelled. 

Satellites  often  operate  for  a  number  of  years  before  they  are  replaced.  The 
replacement  satellite  is  often  the  beneficiary  of  numerous  technological  improvements 
since  the  launch  of  the  previous  satellite.  Therefore,  satellites  most  likely  are  not 
replaced  by  identical  satellites  as  we  have  assumed.  Relaxing  this  assumption  to  allow 
satellites  to  be  replaced  by  non-identical  satellites  would  remove  the  Markov  property 
from  the  decision  problem  and  increase  the  complexity  of  the  model.  However, 
incorporating  this  change  would  greatly  increase  the  model’s  representation  of  the 
real  world. 

We  have  considered  the  number  of  satellites  to  be  constant  throughout  the 
planning  horizon.  However,  if  a  satellite  being  replaced  remains  partially  capable,  it 
may  not  be  removed  from  the  constellation  when  the  replacement  satellite  arrives. 
In  other  words,  a  satellite  replacement  may  actually  be  a  satellite  addition.  This 
implies  that  the  number  of  satellites  in  a  constellation  may  be  dynamic.  Therefore, 
an  interesting  extension  of  this  thesis  is  to  allow  the  number  of  satellites  in  a  con¬ 
stellation  to  change  over  time  depending  on  the  maintenance  action.  One  possible 
method  of  incorporating  this  change  is  to  consider  the  stochastic  evolution  of  a  satel¬ 
lite  constellation  as  a  bivariate  process  in  which  one  random  variable  accounts  for 
the  number  of  satellites  and  the  other  random  variable  describes  the  capability  level 
of  the  constellation.  Including  this  change  would  impact  the  model  by  increasing  the 
dimensionality  of  the  constellation’s  state  space  and  increase  the  likelihood  of  a  state 
space  explosion  when  applied  to  a  large  constellation.  However,  allowing  the  number 
of  satellites  in  the  constellation  to  be  dynamic  would  broaden  the  applicability  of 
the  model  to  more  complex  real  world  systems. 

The  assumption  of  instantaneous  maintenance  is  common  in  the  optimal  main¬ 
tenance  literature  for  general  degrading  systems  because  the  time  to  perform  main- 
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tenance  is  often  negligible  compared  to  the  machine  operating  time.  However,  this 
may  not  be  the  case  for  satellite  constellations.  It  may  take  months  or  even  years  to 
execute  a  maintenance  action  for  a  satellite  constellation  (e.g.,  [16]).  Relaxing  this 
assumption  will  have  the  effect  of  making  the  satellite  degradation  level  transitions 
non-Markovian  after  a  decision  is  made  to  perform  maintenance.  This  results  in 
reducing  the  analytical  tractability  of  the  problem;  however,  relaxing  these  assump¬ 
tions  would  significantly  improve  the  applicability  of  the  model. 

This  research  effort  considered  satellites  whose  function  lifetimes  are  exponen¬ 
tially  distributed;  however,  it  can  be  argued  that  this  is  not  the  case  (e.g.,  [16]  and 
[17]).  Generalizing  the  methodology  to  include  non-exponentially  distributed  life¬ 
times  for  satellite  functions  could  prove  to  be  the  most  fruitful  area  of  improvement. 
If  non-exponential  function  lifetime  distributions  are  incorporated,  the  stochastic 
constellation  degradation  process  may  no  longer  be  modelled  as  a  discrete-time 
Markov  chain.  However,  a  much  less  restrictive  set  of  assumptions  may  be  incor¬ 
porated  to  allow  the  stochastic  degradation  of  the  constellation  to  be  modelled  as 
a  semi-Markov  process.  Putting  the  degradation  process  into  this  framework  allows 
a  stationary  optimal  maintenance  policy  to  be  computed  using  a  semi-Markov  de¬ 
cision  process  as  described  by  Puterman  [32:530].  The  additional  flexibility  in  such 
a  model  may  allow  satellite  constellation  program  managers  to  plan  for  preventive 
maintenance  such  as  preemptive  satellite  replacements.  An  example  of  a  specific 
application  improvement  follows. 

Consider  a  satellite  constellation  containing  a  satellite  whose  operations  are 
extremely  vital  to  national  security.  In  other  words,  if  this  particular  satellite  fails, 
national  security  may  be  compromised.  Under  these  conditions,  it  may  be  beneficial 
to  preemptively  maintain  the  high-value  satellite  and  reduce  the  chance  of  suffering  a 
failure.  When  function  lifetimes  are  assumed  to  be  exponentially  distributed,  failures 
are  truly  random  events;  therefore,  preemptively  maintaining  the  satellite  does  not 
reduce  the  probability  that  it  fails  in  the  next  instant.  As  a  result,  the  optimal  policy 


5-6 


will  call  for  maintenance  of  the  satellite  the  first  time  it  is  observed  to  be  in  the 
failed  state.  However,  if  the  satellite  displays  non-exponentially  distributed  function 
lifetimes  with  increasing  failure  rates  (IFR),  preventively  replacing  the  satellite  will 
decrease  the  probability  of  suffering  a  failure  in  the  next  instant. 

In  conclusion,  optimal  maintenance  policies  for  degrading  satellite  constella¬ 
tions  present  a  number  of  challenges  for  future  research.  Traditional  methods  of 
determining  such  policies  for  general  degrading  systems  have  been  explored  in  great 
detail  in  the  literature;  however  the  inaccessibility  of  the  environment  in  which  satel¬ 
lite  constellations  operate  makes  the  constellation  maintenance  problem  unique.  This 
thesis  provided  further  insight  into  this  problem  by  allowing  each  satellite  in  the  con¬ 
stellation  to  degrade  over  time  leaving  the  constellation  in  a  finite  number  of  partially 
capable  states.  While  this  is  not  the  end-all  solution  to  the  satellite  constellation 
maintenance  problem,  it  serves  as  a  stepping  stone  for  future  research  that  may  pro¬ 
vide  the  fidelity  needed  to  implement  such  procedures  for  the  analysis  of  both  public 
and  private  sector  satellite  constellations. 
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Appendix  A.  One-Satellite  Code 
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7.  Author:  lLt  Tim  Cook 
7.  AFIT/ENS/G0R-05M 

7.  March  2005 

7,  This  program  solves  a  Markov  decision  process  formulated  to  find  an 
7.  optimal  maintenance  policy  for  a  one-satellite  constellation. 

7. 


7,  This  code  assumes  that  the  satellite  will  be  replaced  with  an  identical 
7,  satellite.  It  also  assumes  that  the  action  set  is  stationary  and  that 
7.  the  terminal  reward  is  0  for  all  states. 


7o7.7o7o7o7o7.7o7o7.7o7.7o7o7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o707o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7. 


7,  Define  Parameters 
N  =  20;  7*  Length  of  planning  horizon 
M  =  3;  7o  Number  of  functions 

7,  Total  number  of  states  that  the  satellite  can  possibly  enter  into 
TotStates  =  2~M; 


7,  Number  of  possible  actions 
7,  Action  1:  do  nothing 

7,  Action  2:  on-orbit  repair 

7,  Action  3:  satellite  replacement 

actions  =  3; 

7.  Failure  rates  —  1/ (expected  lifetime  in  years)* (4  quarters) 
lambdal  =  1/(5. 5*4);  7»  Failure  rate  of  function  1 

lambda2  =  1/(5. 25*4);  %  Failure  rate  of  function  2 

lambda3  =  1/(6. 5*4);  l  Failure  rate  of  function  3 

7.  Penalty  costs 

C_p  =  [0  -200  -500  -600  -300  -400  -400  -700] ; 

7,  On  orbit  repair  cost  by  state 

Launch  =  450;  C_m  =  [-realmax  -(Launch  +  20)  -(Launch  +  15) 
-(Launch  +  35)  -(Launch  +  35)  -(Launch  +  55)  -(Launch  +  50) 
-(Launch  +  70)]; 

7,  Satellite  replacement  cost 
C_s  =  -500; 

7,  Use  when  an  action  is  not  feasible  for  that  state 
NA  =  -99999; 
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"/,  Failure  probabilities  for  each  function 
F  =  [l-exp(-lambdal)  1-exp (-lambda2)  1-exp (-lambda3)] ; 

"/,  Survival  probabilities  for  each  function 
R  =  [exp(-lambdal)  exp(-lambda2)  exp(-lambda3)] ; 

"/,  Probability  of  a  successful  repair  for  each  function 
M  =  [.95  .96  .97] ; 

"/,  Probability  of  a  successful  replacement  for  each  function 
G  =  [.975  .94  .98] ; 
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I  Define  Transition  Probabilities 

l  p(s,a,j)  ==>  Probability  of  going  to  state  j,  given  presently  in 
l  state  s  and  choosing  action  a 

p  =  zeros (TotStates ,  actions,  TotStates); 


p(l,l,l)  =  R(1)*R(2)*R(3) 
p(l , 1 ,2)  =  R(1)*R(2)*F(3) 
p(l , 1 ,3)  =  R(1)*F(2)*R(3) 
p(l , 1 ,4)  =  R(1)*F(2)*F(3) 
p(l , 1 ,5)  =  F(1)*R(2)*R(3) 
p(l , 1 ,6)  =  F(1)*R(2)*F(3) 
p(l , 1 ,7)  =  F(1)*F(2)*R(3) 
p(l , 1 ,8)  =  F(1)*F(2)*F(3) 


l  There  is  no  need  to  include  the  transition  probabilities  for  action  2 
l  while  in  state  2  because  the  action  will  never  be  chosen.  This  is  because 
l  the  transition  probabilities  are  identical  to  those  of  action  1,  but  the 
l  large  setup  cost  is  not  incurred  with  action  1. 


p(l ,3 , 1) 
p(l ,3 ,2) 
p(l ,3 ,3) 
p(l ,3 ,4) 
p(l ,3 ,5) 
p(l,3,6) 
p(l ,3 ,7) 
p(l ,3 ,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3>)  ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)> ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


p(2 , 1 ,2)  =  R(1)*R(2); 
p(2 , 1 ,4)  =  R(1)*F(2); 
p(2,l,6)  =  F(1)*R(2); 
p(2 , 1 ,8)  =  F(1)*F(2); 
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p(2 ,2 , 1) 
p(2,2,2) 

p(2,2,3) 

p(2,2,4) 

p(2,2,5) 

p(2,2,6) 

p(2,2,7) 

p(2,2,8) 

p(2,3,l) 

p(2,3,2) 

p(2,3,3) 

p(2,3,4) 

p(2,3,5) 

p(2,3,6) 

p(2,3,7) 

p(2,3,8) 

p(3,l,3) 
p(3,l,4) 
p(3, 1 ,7) 
p(3 , 1 ,8) 

p(3,2 , 1) 
p(3,2,2) 
p(3,2,3) 
p(3,2,4) 
p(3,2,5) 
p(3,2,6) 
p(3,2,7) 
p(3,2,8) 

p(3,3,l) 

p(3,3,2) 

p(3,3,3) 

p(3,3,4) 

p(3,3,5) 

p(3,3,6) 

p(3,3,7) 

p(3,3,8) 

p(4,l,4) 


R(1)*R(2)*M(3) ; 
R(1)*R(2)*(1-M(3)) ; 
R(1)*F(2)*M(3) ; 
R(1)*F(2)*(1-M(3)) ; 
F(1)*R(2)*M(3) ; 
F(1)*R(2)*(1-M(3)) ; 
F(1)*F(2)*M(3) ; 
F(1)*F(2)*(1-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3))  ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(l-G(l) ) *G(2) *G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 

( 1— G ( 1) ) * (1-G(2) )*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 

R(1)*R(3) ; 

R(1)*F(3) ; 

F(1)*R(3) ; 

F(1)*F(3) ; 

R(1)*M(2)*R(3) ; 
R(1)*M(2)*F(3) ; 
R(1)*(1-M(2))*R(3) ; 
R(1)*(1-M(2))*F(3) ; 
F(1)*M(2)*R(3) ; 
F(1)*M(2)*F(3) ; 
F(1)*(1-M(2))*R(3) ; 
F(1)*(1-M(2))*F(3) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(l); 


A-3 


p(4,l,8)  =  F(l); 


p(4,2,l) 

p(4,2,2) 

p(4,2,3) 

p(4,2,4) 

p(4,2,5) 

p(4,2,6) 

p(4,2,T) 

p(4,2,8) 


R(1)*M(2)*M(3) ; 
R(1)*M(2)*(1-M(3)) ; 
R(1)*(1-M(2))*M(3) ; 
R(l)*(l-M(2))*(l-M(3)) ; 
F(1)*M(2)*M(3) ; 
F(1)*M(2)*(1-M(3)) ; 
F(1)*(1-M(2))*M(3) ; 
F(l)*(l-M(2))*(l-M(3)) ; 


p(4,3,l) 

p(4,3,2) 

p(4,3,3) 

p(4,3,4) 

p(4,3,5) 

p(4,3,6) 

p(4,3,7) 

p(4,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


p(5 , 1 ,5)  =  R(2)*R(3); 
p(5,l,6)  =  R(2)*F(3); 
p(5 , 1 , 7)  =  F(2)*R(3); 
p(5,l,8)  =  F (2) *F (3) ; 


p(5 ,2 , 1) 
p(5,2,2) 
p(5,2,3) 
p(5,2,4) 
p(5,2,5) 
p(5,2,6) 
p(5,2,7) 
p(5,2,8) 


M(1)*R(2)*R(3) ; 
M(1)*R(2)*F(3) ; 
M(1)*F(2)*R(3) ; 
M(1)*F(2)*F(3) ; 
(1-M(1))*R(2)*R(3) 
(1-M(1))*R(2)*F(3) 
(1-M(1))*F(2)*R(3) 
(1-M(1))*F(2)*F(3) 


p(5,3,l) 

p(5,3,2) 

p(5,3,3) 

p(5,3,4) 

p(5,3,5) 

p(5,3,6) 

p(5,3,7) 

p(5,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 
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p(6, 1 ,6) 
p(6 , 1 ,8) 

p(6,2,l) 

p(6,2,2) 

p(6,2,3) 

p(6,2,4) 

p(6,2,5) 

p(6,2,6) 

p(6,2,7) 

p(6,2,8) 

p(6,3,l) 

p(6,3,2) 

p(6,3,3) 

p(6,3,4) 

p(6,3,5) 

p(6,3,6) 

p(6,3,7) 

p(6,3,8) 

p(7 , 1 ,7) 
p(7,l,8) 

p(7 ,2 , 1) 
p(7,2,2) 
p(7,2,3) 
p(7,2,4) 
p(7,2,5) 
p(7 ,2,6) 
p(7 ,2,7) 
p(7,2,8) 

p(7 ,3,1) 
p(7,3,2) 
p(7,3,3) 
p(7,3,4) 
p(7,3,5) 
p(7,3,6) 
p(7 ,3,7) 
p(7,3,8) 

p(8,l,8) 


R(2); 
F (2)  ; 


M(1)*R(2)*M(3) ; 
M(1)*R(2)*(1-M(3)) ; 
M(1)*F(2)*M(3) ; 
M(1)*F(2)*(1-M(3)) ; 
(1-M(1))*R(2)*M(3) ; 
(1-M(1))*R(2)*(1-M(3)) ; 
(1-M(1))*F(2)*M(3) ; 
(1-M(1))*F(2)*(1-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(1-G(1))*(1-G(2))*G(3); 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(3); 
F (3) ; 


M(1)*M(2)*R(3) ; 
M(1)*M(2)*F(3) ; 
M(1)*(1-M(2))*R(3) ; 
M(1)*(1-M(2))*F(3) ; 
(1-M(1))*M(2)*R(3) ; 
(1-M(1))*M(2)*F(3) ; 
(1-M(1))*(1-M(2))*R(3) ; 
(1-M(1))*(1-M(2))*F(3) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(1-G(1))*(1-G(2))*G(3) ; 

( 1— G ( 1) ) * (1-G(2) )*(1-G(3) ) ; 

1; 
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p(8 ,2 , 1) 
p(8,2,2) 

p(8,2,3) 

p(8,2,4) 

p(8,2,5) 

p(8,2,6) 

p(8,2,7) 

p(8,2,8) 


M(1)*M(2)*M(3) ; 
M(1)*M(2)*(1-M(3)) ; 
M(1)*(1-M(2))*M(3) ; 
M(l)*(l-M(2))*(l-M(3)) ; 
(1-M(1))*M(2)*M(3) ; 
(1-M(1))*M(2)*(1-M(3)) ; 
(1-M(1))*(1-M(2))*M(3) ; 
(1-M(1))*(1-M(2))*(1-M(3)) ; 


p(8,3,l)  =  G(1)*G(2)*G(3) ; 
p(8,3,2)  =  G(1)*G(2)*(1-G(3)) ; 
p(8,3,3)  =  G(1)*(1-G(2))*G(3) ; 
p(8,3,4)  =  G(l)*(l-G(2))*(l-G(3)) ; 
p(8,3,5)  =  (l-G(l) )*G(2)*G(3) ; 
p(8,3,6)  =  (l-G(l) ) *G(2) * (1-G(3) ) ; 
p(8,3,7)  =  ( 1— G ( 1) ) * (1-G(2) )*G(3) ; 
p(8,3,8)  =  (1-G(1))*(1-G(2))*(1-G(3)); 
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I  Define  the  Immediate  Rewards  -  deterministic  part  of  the  reward  function 
l  ra{a}(s)  ==>  Immediate  reward  when  choosing  action  a  in  state  s 


ra  =  cell (actions) ; 
ra{l}  =  zeros (TotStates) 
ra{2}  =  zeros (TotStates) 
ra{3}  =  zeros (TotStates) 


ra{2} (1) 

=  NA; 

ra{2>(2) 

=  C_m(2) ; 

ra{2>(3) 

=  C_m(3) ; 

ra{2>(4) 

=  C_m(4); 

ra{2>(5) 

=  C_m(5) ; 

ra{2>(6) 

=  C_m(6) ; 

ra{2}(7) 

=  C_m(7); 

ra{2>(8) 

=  C_m(8) ; 

ra{3} (1) 

=  C_s; 

ra{3>(2) 

=  C_s; 

ra{3>(3) 

=  C_s ; 

ra{3}(4) 

=  C_s; 

ra{3>(5) 

=  C_s; 

ra{3>(6) 

=  C_s; 
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ra{3}(7)  =  C_s; 
ra{3}(8)  =  C_s; 


7.  Define  the  Expected  Rewards 

7*  r(s,a)  ==>  The  expected  reward  when  in  state  s  and  choosing  action  a. 

7,  This  is  equivalent  to  the  immediate  reward  plus  the  expected  penalty  cost 
7,  over  the  next  period, 
r  =  zeros (TotStates, actions) ; 

for  s  =  l:TotStates 
for  a  =  1: actions 

7.  Calculate  the  expected  penalty  cost  over  the  next  period  -  the 
7,  sum  of  the  penalty  costs  times  the  probability  that  that  cost  is 
7,  incurred. 

Expected_Penalty  =  0; 
for  j  =  1: TotStates 

Expected_Penalty  =  Expected_Penalty  +  p(s,a, j)*C_p(j) ; 

end 

7*  Assign  the  expected  reward  resulting  from  taking  action  a  in 
7,  state  s . 

if  Expected_Penalty  ==  0 

r(s,a)  =  NA;  %  The  action  is  not  feasible 

else 

r(s,a)  =  ra{a}(s)  +  Expected_Penalty ; 

end 

end 

end 

7,  Backward  Induction  Code  (Directly  adapted  from  Capt  Sumter’s  code) 

7,  Define  the  u,  ustar  and  dstar  vector 
u  =  zeros (TotStates , actions, N) ; 
ustar  =  zeros (TotStates ,N) ; 
dstar  =  zeros (TotStates ,N) ; 

7.  Note:  The  value  of  any  action  in  the  final  time  period  has  zero 
7,  reward  and  therefore  zero  utility:  ustar (s,N)  =  0  for  all  s 

n  =  N  -  1;  while  n  >=  1 
for  s  =  1: TotStates 

ustar (s,n)  =  -realmax; 
for  a  =  1: actions 

if  r(s,a)  ~=  NA  l  Infeasible  action,  reward  not  applicable 
expected_value  =  0; 
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for  j  =  l:TotStates 

expected_value  =  expected_value+p(s,a, j)*ustar(j ,n+l) ; 

end 

u(s,a,n)  =  r(s,a)  +  expected_value ; 
if  u(s,a,n)  >  ustar(s,n) 
ustar(s,n)  =  u(s,a,n); 
dstar(s,n)  =  a; 

end 


end 

end 

end 

n  =  n  -  1;  %  Decrement  the  time 

end 


7*  Display  the  results  to  the  screen 
disp( ’Optimal  Policies’); 
disp(dstar) ; 

disp(’Values  of  ustar  given  the  starting  state’); 
disp(ustar) ; 


7,  Open  file  in  current  directory  to  write  results  to 
fid  =  fopen(’ One_Sat_Solution.txt’ , ’w’) ; 


7,  Write  the  results  to  the  file 

fprintf (fid,  ’Optimal  Policy’);  fprintf(fid,  ’\n’);  for  i  = 
1 :TotStates ; 
for  j=l:N 

fprintf  (fid,  ’7»g\t’  ,dstar(i,  j)) ; 

end 


fprintf (fid,  ’\n’); 

end  fprintf (fid,  ’\n  Optimal  Values  \n’);  for  i  =  l:TotStates; 
for  j=l:N 

fprintf  (fid,  ’7«8 . 5f  \t  ’  ,  ustar  (i ,  j  )  )  ; 

end 


fprintf (fid,  ’\n’); 

end 


7,  Close  file 
f close(f id) ; 
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Appendix  B.  Three-Satellite  Code 


7:/.y.y//:/.7.y:/:/.y.7.y:/:/.y.7:/:/.7.y:/:/.y.y:/:/:/.y.7:/;/.7.y:/:/.y.7:/;/.7.y.y:/;/.7.y:/:/.y.y:/:/:/.7.,/:/.7.y.7:/:/.7.,/.7.,/.y.y.,/:/.7:/.y.7. 


7.  Author:  lLt  Tim  Cook 
7.  AFIT/ENS/G0R-05M 

7.  March  2005 

7,  This  program  solves  a  Markov  decision  process  formulated  to  find  an 
7.  optimal  maintenance  policy  for  a  three  satellite  constellation  in  which 
7,  the  first  and  second  satellite  have  2  functions  and  the  third  satellite 
7,  has  3  functions. 

7. 


7,  This  code  assumes  that  a  satellite  will  be  replaced  with  an  identical 
7.  satellite.  It  also  assumes  that  the  action  set  is  stationary  and  that 
7,  the  terminal  reward  is  0  for  all  states.  This  solution  also  assumes  that 
7,  only  one  satellite  can  be  launched  on  a  single  launch  vehicle,  but 
7.  multiple  satellites  may  be  replaced  in  one  time  step. 


7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7.7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7.7o 


7,  Define  Parameters 

N  =  20;  7o  Length  of  planning  horizon 

M  =  [2  2  3];  7.  Number  of  functions  desired  operational  in  each  satellite 

K  =  3;  7.  Number  of  Satellites 

7,  Create  a  vector  that  keeps  the  number  of  degradation  levels  for  each 
7,  satellite 

states  =  zeros (K);  for  i=l:K 

states(i)  =  2~M(i) ;  7#  Number  of  possible  states  for  satellite  i 

end 

7,  Define  the  total  number  of  states  that  the  entire  satellite  constellation 
7,  can  possibly  enter  into 
TotStates  =  prod(states ( : , 1) ) ; 

7,  Number  of  possible  actions  for  each  satellite 
actions  =  3; 

7.  Failure  rates  —  1/ (expected  lifetime  in  years)* (4  quarters) 
lambdal  =  1/(5. 5*4);  %  Failure  rate  of  function  1 

lambda2  =  1/(5. 25*4);  l  Failure  rate  of  function  2 

lambda3  =  1/(6. 5*4);  %  Failure  rate  of  function  3 

7,  On  Orbit  Repair  Cost  by  state  for  each  satellite 

C_m  =  cell(K); 
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LaunchVehicle  =  -450; 

C_m{l}  =  [-realmax  LaunchVehicle-15  LaunchVehicle-35 
C_m{2}  =  [-realmax  LaunchVehicle-15  LaunchVehicle-35 
C_m{3}  =  [-realmax  LaunchVehicle-20  LaunchVehicle-15 
LaunchVehicle-35  LaunchVehicle-55  LaunchVehicle-50 


LaunchVehicle-50] ; 
LaunchVehicle-50] ; 
LaunchVehicle-35 
LaunchVehicle-70] ; 


7.  Satellite  Replacement  costs 
C_s  =  [-500  -500  -520] ; 

7.  Use  when  an  action  is  not  feasible  for  that  state 
NA  =  -99999; 


7,  Failure  probabilities  for  each  function 
F  =  [1-exp (-lambdal)  1-exp (-lambda2)  1-exp (-lambda3)] ; 

7,  Survival  probabilities  for  each  function 
R  =  [exp (-lambdal)  exp(-lambda2)  exp(-lambda3)] ; 

7,  Probability  of  a  successful  repair  for  each  function 
M  =  [.95  .96  .97] ; 

7,  Probability  of  a  successful  replacement  for  each  function 
G  =  [.975  .94  .98] ; 


7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7.7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7o7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7.7o 


7.  Define  Transition  Probabilities  for  individual  satellites 
7.  p{k}(s,a,j)  ==>  Probability  for  satellite  k  of  going  to  state  j,  given 
7.  presently  in  state  s  and  choosing  action  a  for  that 

7.  individual  satellite 


7.  Allocate  memory  for  the  transition  probability  matricies 
7.  using  cell  arrays 
p  =  cell(K);  for  i  =  1:K 

p(i)  =  {zeros(states(i) ,  actions,  states(i))}; 

end 


7.  Define  the  transition  probabilities 
p{l}(l, 1, 1)  =  R(l) *R(2) ; 
p{l}(l,l,2)  =  R(l) *F (2) ; 
p{l}(l,l,3)  =  F(1)*R(2); 

P{1}(1,1,4)  =  F(l) *F (2) ; 

7.  There  is  no  need  to  include  the  transition  probabilities  for  action  2 
7.  while  in  state  2  because  the  action  will  never  be  chosen.  This  is  because 
7.  the  transition  probabilities  are  identical  to  those  of  action  1,  but  the 
7.  large  setup  cost  is  not  incurred  with  action  1.  The  same  holds  for  the  other 
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'/,  two  satellites. 


p{l}(l,3,l) 
p{l}(l , 3 ,2) 
p{l}(l,3,3) 
p{l>(l,3,4) 


G(1)*G(2) ; 

G(l) *(1-G(2) ) ; 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)) 


p{l} (2 , 1 , 2)  =  R(l); 
p{l} (2,1,4)  =  F(l) ; 


p{l>(2,2,l) 

p{l>(2,2,2) 

p{l>(2,2,3) 

p{l>(2,2,4) 


R(1)*M(2) ; 
R(l)*(l-M(2))  ; 
F(1)*M(2) ; 

F(l) *(1-M(2) ) ; 


p{l>(2,3,l) 

p{l>(2,3,2) 

p{l}(2,3,3) 

p{l>(2,3,4) 


G(1)*G(2) ; 
G(l)*(l-G(2)) ; 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)) 


p{l} (3 , 1 , 3)  =  R(2) ; 
p{l} (3,1,4)  =  F(2) ; 


p{l>(3,2,l) 

p{l>(3,2,2) 

p{l>(3,2,3) 

p{l>(3,2,4) 


M(1)*R(2) ; 
M(1)*F(2) ; 
(l-M(l) ) *R(2) ; 
(1-M(1))*F(2); 


p{l}(3,3,l) 

p{l>(3,3,2) 

p{l>(3,3,3) 

p{l>(3,3,4) 


G(1)*G(2) ; 
G(l)*(l-G(2)); 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)) 


p{l} (4, 1 , 4)  =  1; 


p{l>(4,2,l) 

p{l>(4,2,2) 

p{l>(4,2,3) 

p{l>(4,2,4) 


M(1)*M(2) ; 
M(l)*(l-M(2))  ; 
(l-M(l) ) *M(2) ; 
(1-M(1))*(1-M(2)) 


p{l>(4,3,l) 

p{l>(4,3,2) 

p{l>(4,3,3) 


G(1)*G(2) ; 

G(l)*(l-G(2)); 

(1-G(1))*G(2); 


p{l}(4,3,4)  =  (1-G(1))*(1-G(2)) ; 


p{2}(l,l,l)  =  R(1)*R(2); 
p{2} (1 ,1,2)  =  R(l) *F (2) ; 
p{2} (1 ,1,3)  =  F(l) *R(2) ; 
p{2}( 1,1,4)  =  F(1)*F(2); 


p{2} (1 ,3,1)  =  G(1)*G(2); 
p{2} (1 ,3,2)  =  G(l) *(1-G(2) ) ; 
p{2} (1 ,3,3)  =  (1-G(1))*G(2); 
p{2} (1 ,3,4)  =  (1-G(1))*(1-G(2)); 

p{2} (2,1,2)  =  R(l); 
p{2} (2,1,4)  =  F(l) ; 

p{2} (2,2,1)  =  R(1)*M(2); 
p{2}(2,2,2)  =  R(l)*(l-M(2)); 
p{2}(2,2,3)  =  F(1)*M(2); 
p{2} (2,2,4)  =  F(l) *(1-M(2) ) ; 

p{2} (2,3,1)  =  G(1)*G(2) ; 
p{2}(2,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{2}(2,3,3)  =  (1-G(1))*G(2) ; 
p{2}(2,3,4)  =  (1-G(1))*(1-G(2)); 

p{2} (3,1,3)  =  R(2) ; 
p{2} (3,1,4)  =  F (2) ; 

p{2} (3,2,1)  =  M(1)*R(2); 
p{2} (3,2,2)  =  M(1)*F(2); 
p{2} (3,2,3)  =  (l-M(l) ) *R(2) ; 
p{2}(3,2,4)  =  (1-M(1))*F(2); 

p{2} (3,3,1)  =  G(1)*G(2) ; 
p{2}(3,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{2}(3,3,3)  =  (1-G(1))*G(2); 
p{2}(3,3,4)  =  (1-G(1))*(1-G(2)); 

p{2} (4,1,4)  =  1; 

p{2}(4,2, 1)  =  M(1)*M(2); 
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p{2}(4,2,2)  =  M(l)*(l-M(2)) ; 
p{2}(4,2,3)  =  (1-M(1))*M(2) ; 
p{2}(4,2,4)  =  (1-M(1))*(1-M(2)); 


p{2>(4,3,l)  =  G(1)*G(2); 
p{2}(4,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{2}(4,3,3)  =  ( 1— G ( 1) ) *G (2) ; 
p{2}(4,3,4)  =  (1-G(1))*(1-G(2)) ; 


mfflnfflnnfflfflnmmmfflmmfflfflranfflfflmmmnfflm 


p{3} (1 ,1,1)  =  R(1)*R(2)*R(3) ;  p{3} (1 , 1 ,2)  =  R(l) *R(2) *F(3) 
p{3} (1 ,1,3)  =  R(1)*F(2)*R(3) ;  p{3} (1 , 1 ,4)  =  R(l) *F(2) *F(3) 
p{3} (1 ,1,5)  =  F(1)*R(2)*R(3) ;  p{3}(l,l,6)  =  F(l) *R(2) *F(3) 


p{3} (1 ,1,7)  =  F(1)*F(2)*R(3) ;  p{3} (1 , 1 , 8)  =  F(l) *F(2) *F(3) 


p{3} (1 ,3,1)  =  G(1)*G(2)*G(3); 
p{3>(l,3,2)  =  G(1)*G(2)*(1-G(3)) ; 
p{3} (1 ,3,3)  =  G(1)*(1-G(2))*G(3); 


p{3}(l,3,4)  =  G(l)*(l-G(2))*(l-G(3)); 
p{3}(l,3,5)  =  (1-G(1))*G(2)*G(3) ; 
p{3}(l,3,6)  =  ( 1— G ( 1) ) *G (2) *(1— G(3)) ; 
p{3}(l,3,7)  =  (1-G(1))*(1-G(2))*G(3) ; 
p{3}(l,3,8)  =  (1-G(i))*(i-G(2))*(l-G(3))j 


p{3}(2,l,2)  =  R(1)*R(2); 
p{3} (2,1,4)  =  R(1)*F(2); 
p{3} (2,1,6)  =  F(1)*R(2); 
p{3} (2,1,8)  =  F(1)*F(2); 


p{3} (2,2,1)  =  R(1)*R(2)*M(3) ; 
p{3} (2,2,2)  =  R(1)*R(2)*(1-M(3)) ; 


p{3} (2,2,3)  =  R(1)*F(2)*M(3); 
p{3}(2,2,4)  =  R(1)*F(2)*(1-M(3)) ; 
p{3}(2,2,5)  =  F(1)*R(2)*M(3) ; 
p{3}(2,2,6)  =  F(1)*R(2)*(1-M(3)) ; 
p{3} (2,2,7)  =  F(1)*F(2)*M(3); 
p{3}(2,2,8)  =  F(1)*F(2)*(1-M(3)) ; 


p{3}(2,3,l)  =  G(1)*G(2)*G(3); 
p{3}(2,3,2)  =  G(1)*G(2)*(1-G(3)); 
p{3}(2,3,3)  =  G(1)*(1-G(2))*G(3); 
p{3}(2,3,4)  =  G(l)*(l-G(2))*(l-G(3)); 
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p{3}(2,3,5) 

p{3}(2,3,6) 

p{3}(2,3,7) 

p{3}(2,3,8) 

p{3} (3,1,3) 
p{3} (3,1,4) 
p{3} (3,1,7) 
p{3} (3,1,8) 

p{3}(3,2,l) 
p{3}(3,2,2) 
p{3}(3,2,3) 
p{3}(3,2,4) 
p{3}(3,2,5) 
p{3}(3,2,6) 
p{3}(3,2,7) 
p{3} (3,2,8) 

p{3} (3,3,1) 
p{3}(3,3,2) 
p{3}(3,3,3) 
p{3}(3,3,4) 
p{3}(3,3,5) 
p{3}(3,3,6) 
p{3}(3,3,7) 
p{3}(3,3,8) 

p{3} (4,1,4) 
p{3} (4,1,8) 

p{3}(4,2, 1) 
p{3}(4,2,2) 
p{3}(4,2,3) 
p{3}(4,2,4) 
p{3}(4,2,5) 
p{3}(4,2,6) 
p{3}(4,2,7) 
p{3}(4,2,8) 

p{3} (4,3,1) 
p{3}(4,3,2) 
p{3}(4,3,3) 


(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3); 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

R(l) *R(3) ; 

R(1)*F(3) ; 

F(1)*R(3) ; 

F(1)*F(3) ; 

R(l) *M(2) *R(3) ; 

R(l) *M(2) *F (3) ; 
R(1)*(1-M(2))*R(3) ; 
R(1)*(1-M(2))*F(3) ; 

F(l) *M(2) *R(3) ; 

F(l) *M(2) *F (3) ; 
F(1)*(1-M(2))*R(3) ; 
F(1)*(1-M(2))*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(l); 

F  ( 1 ) ; 


R(1)*M(2)*M(3) ; 
R(1)*M(2)*(1-M(3)) ; 
R(1)*(1-M(2))*M(3) ; 
R(l)*(l-M(2))*(l-M(3)) ; 
F(1)*M(2)*M(3) ; 
F(1)*M(2)*(1-M(3)) ; 
F(1)*(1-M(2))*M(3) ; 
F(l)*(l-M(2))*(l-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
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p{3}(4,3,4) 

p{3}(4,3,5) 

p{3}(4,3,6) 

p{3}(4,3,7) 

p{3}(4,3,8) 

p{3}(5,l,5) 
p{3} (5,1,6) 
p{3} (5,1,7) 
p{3} (5,1,8) 

p{3} (5,2,1) 
p{3}(5,2,2) 
p{3}(5,2,3) 
p{3}(5,2,4) 
p{3}(5,2,5) 
p{3}(5,2,6) 
p{3}(5,2,7) 
p{3}(5,2,8) 

p{3}(5,3,l) 

p{3}(5,3,2) 

p{3}(5,3,3) 

p{3}(5,3,4) 

p{3}(5,3,5) 

p{3}(5,3,6) 

p{3}(5,3,7) 

p{3}(5,3,8) 

p{3} (6,1,6) 
p{3} (6,1,8) 

p{3} (6,2,1) 
p{3}(6,2,2) 
p{3}(6,2,3) 
p{3} (6,2,4) 
p{3} (6,2,5) 
p{3}(6,2,6) 
p{3}(6,2,7) 
p{3}(6,2,8) 

p{3} (6,3,1) 
p{3}(6,3,2) 


G(l)*(l-G(2))*(i-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

R(2)*R(3)  ; 

R(2) *F (3) ; 

F (2) *R(3) ; 

F (2) *F (3) ; 

M(l) *R(2) *R(3) ; 

M(l) *R(2) *F (3) ; 

M(l) *F (2) *R(3) ; 

M(l) *F (2) *F (3) ; 
(1-M(1))*R(2)*R(3) ; 
(1-M(1))*R(2)*F(3) ; 
(1-M(1))*F(2)*R(3) ; 
(1-M(1))*F(2)*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(l))*(i-G(2))*G(3) ; 
(1-G(l))*(i-G(2))*(l-G(3)); 


R(2); 
F(2)  ; 


M(1)*R(2)*M(3) ; 
M(1)*R(2)*(1-M(3)) ; 
M(1)*F(2)*M(3) ; 
M(1)*F(2)*(1-M(3)) ; 
(1-M(1))*R(2)*M(3) ; 
(1-M(1))*R(2)*(1-M(3)) ; 
(1-M(1))*F(2)*M(3) ; 
(1-M(1))*F(2)*(1-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
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p{3}(6,3,3) 

p{3}(6,3,4) 

p{3}(6,3,5) 

p{3}(6,3,6) 

p{3}(6,3,7) 

p{3}(6,3,8) 

p{3} (7,1,7) 
p{3} (7,1,8) 

p{3}(7 ,2,1) 
p{3}(7 ,2,2) 
p{3}(7 ,2,3) 
p{3}(7 ,2,4) 
p{3}(7 ,2,5) 
p{3}(7,2,6) 
p{3}(7 ,2,7) 
p{3} (7,2,8) 

p{3}(7,3,l) 
p{3}(7,3,2) 
p{3}(7 ,3,3) 
p{3} (7,3,4) 
p{3}(7,3,5) 
p{3}(7,3,6) 
p{3}(7,3,7) 
p{3}(7 ,3,8) 

p{3} (8,1,8) 

p{3} (8,2,1) 
p{3} (8,2,2) 
p{3}(8,2,3) 
p{3}(8,2,4) 
p{3}(8,2,5) 
p{3}(8,2,6) 
p{3} (8,2,7) 
p{3}(8,2,8) 

p{3} (8,3,1) 
p{3}(8,3,2) 
p{3}(8,3,3) 
p{3}(8,3,4) 


G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3)  ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(3); 
F (3)  ; 


M(1)*M(2)*R(3) ; 
M(1)*M(2)*F(3) ; 
M(1)*(1-M(2))*R(3) ; 
M(1)*(1-M(2))*F(3) ; 
(1-M(1))*M(2)*R(3) ; 
(1-M(1))*M(2)*F(3) ; 
(1-M(1))*(1-M(2))*R(3) ; 
(1-M(1))*(1-M(2))*F(3) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(l-G(l) ) *G(2) *G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(l))*(i-G(2))*(l-G(3))j 

1; 

M(1)*M(2)*M(3) ; 
M(1)*M(2)*(1-M(3)) ; 
M(1)*(1-M(2))*M(3) ; 
M(l)*(l-M(2))*(l-M(3)) ; 
(1-M(1))*M(2)*M(3) ; 

(l-M(l) ) *M(2) * (1-M(3) ) ; 
(1-M(1))*(1-M(2))*M(3) ; 
(1-M(1))*(1-M(2))*(1-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(i-G(3)); 
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p{3}(8,3,5) 

p{3}(8,3,6) 

p{3}(8,3,7) 

p{3}(8,3,8) 


(l-G(l) ) *G(2) *G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


%7.7oyoy//o7.%7o7.%7.0/.yo7.%%7o7o7.%%7.7oyo0/o%7.7oyo0/o%7.0/.yo%%7.%7o7.%%%7o7.%7.7.7o0/.y.0/o7//.%y.7.%7//o7o%%7.7.7oyo0/o%7//o7o7.% 


"/,  Define  the  state  space  for  each  satellite 
Sat_s  -  cell(K);  Sat_s{l}  =[11 
1  0 


0  1 


0  0]  ; 

Sat_s{2>  =[11 
1  0 
0  1 


0  0]  ; 

Sat_s{3>  =[111 
110 
10  1 
10  0 
Oil 
0  10 
0  0  1 
0  0  0]; 


%7.7o7.yo%7.7.y.7o%7.7.yo7o%%7.7o7.7o%7.7oyoyo7.7.7oyoyo%7.7o7o7.%%7.7oy.%%7.7o%yo%7.7o7.yo7o%7.7o7oy.%%7o7o%yo%7.7o7.7o%%7.7o%7o 


I  Define  the  overall  state  space  for  the  constellation 
Constellation_s  =  cell(TotStates) ; 
si  =  1; 
s2  =  1; 
s3  =  1; 

for  i  =  l:TotStates 

Constellation_s{i}  =  [si  s2  s3] ; 


s3  =  s3  +  1; 

if  s3  >  states (3) 
s3  =  1; 
s2  =  s2  +  1; 

end 

if  s2  >  states (2) 
s2  =  1; 
si  =  si  +  1; 

end 

if  si  >  states(l) 
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si  =  1; 


end 


end 
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7,  Penalty  Cost  assigned  per  state 
%  The  decision  maker  desires  that 
C_p  =  zeros (TotStates) ; 
for  i  =  1: TotStates 


7.  Function  1  Penalty  Cost 
si  =  Constellation_s{i}(l) ; 
s2  =  Constellation_s{i}(2) ; 
s3  =  Constellation_s{i}(3) ; 

sum  =  Sat_s{l}(sl , 1)+Sat_s{2}(s2, 1)+Sat_s{3}(s3, 1) ; 
if  sum  ==  3 

C.p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  2 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  1 

C_p(i)  =  C_p(i)  -  500; 
elseif  sum  ==  0 

C_p(i)  =  C_p(i)  -  650; 

end 


7.  Function  2  Penalty  Cost 

sum  =  Sat_s{l}(sl ,2)+Sat_s{2}(s2,2)+Sat_s{3}(s3,2) ; 
if  sum  ==  3 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  2 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  1 

C_p(i)  =  C_p(i)  -  450; 
elseif  sum  ==  0 

C_p(i)  =  C_p(i)  -  700; 

end 


7.  Function  3  Penalty  Cost 
if  Sat_s{3}(s3,3)  ==  1 
C_p(i)  =  C_p(i)  +  0; 
elseif  Sat_s{3}(s3,3)  ==  0 
C_p(i)  =  C_p(i)  -  200; 

end 

end 
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1  Define  the  overall  constellation  action  set  in  a  cell  array 

Constellation_a  =  cell (actions'^) ; 

al=l ; 

a2=l ; 

a3=l ; 

for  i  =  1: actions'^ 

Constellation_a{i}= [al  a2  a3]  ; 

a3  =  a3  +  1 ; 

if  a3  >  actions 
a2  =  a2  +  1; 
a3  =  1; 

end 

if  a2  >  actions 
al  =  al  +  1; 
a2  =  1; 

end 

end 


#/«  Define  the  overall  constellation  transition  probabilities.  Do  this  by 
°/«  looping  through  each  possibile  (s,a,j)  combination  in  the  constellation 
°/«  and  multiplying  the  individual  satellite  transition  probabilities 
7,  together  that  would  lead  to  the  overall  satellite  constellation 
%  transition.  This  is  where  the  vector  notation  for  each  state  comes  in 
#/«  handy.  Also,  the  user  of  this  code  must  know  clearly  what  state  the 
°/«  index  of  each  satellite’s  state  and  action  vector  represents. 


si  =  1; 
s2  =  1; 
s3  =  1; 
al  =  1; 
a2  =  1; 
a3  =  1; 
jl  =  1; 
j2  =  1; 
j3  =  1; 

Constellation_p  =  zeros (TotStates ,actions~K, TotStates) ; 


for  s  = 
for 


1 : TotStates 
a  =  1 : actions'^ 
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for  j  =  l:TotStates 


Constellation_p(s,a, j)  =  p{l}(sl,al, jl)*p{2}(s2,a2, j2) *p{3}(s3,a3, j3) 

“/*  Eliminate  some  transition  probabilities  because  the  actions 
°/*  are  not  feasible.  This  sets  the  transition  probabilities  to 
1  0  for  all  infeasible  actions  and  therefore  sets  the  expected 
°/«  rewards  to  0  so  they  can  be  identified  and  then  set  to  NA. 
if  si  ==  1 

if  al  ==  2 

Constellation_p(s ,a, j)  =  0; 

end 

end 

if  s2  ==  1 

if  a2  ==  2 

Constellation_p(s , a, j)  =  0; 

end 

end 

if  s3  ==  1 

if  a3  ==  2 

Constellation_p(s ,a, j)  =  0; 

end 

end 

°/*  The  previous  block  eliminates  any  action  to  do  an  on  orbit 
°/«  repair  to  a  satellite  that  has  nothing  broken. 

if  si  ==  states(l) 

if  s2  ==  states (2) 

if  s3  ==  states (3) 
if  al  ==  1 

if  a2  ==  1 

if  a3  ==  1 

Constellation_p(s,a, j)  =  0; 

end 

end 

end 

end 

end 

end 

%  The  previous  block  eliminates  the  possibility  of  doing 
“/*  nothing  when  all  functions  of  all  satellites  are 
°/0  non-operational . 
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j3  =  j3  +  1; 

if  j3  >  states(3) 
j3  =  1; 
j2  =  j2  +  1; 

end 

if  j2  >  states(2) 
j2  =  1; 
jl  =  jl  +  1; 

end 

if  jl  >  states(l) 

jl  =  l; 
a3  =  a3  +  1 ; 

end 

if  a3  >  actions 
a3  =  1; 
a2  =  a2  +  1; 

end 

if  a2  >  actions 
a2  =  1; 
al  =  al  +  1; 

end 

if  al  >  actions 
al  =  1; 
s3  =  s3  +  1 ; 

end 

if  s3  >  states (3) 
s3  =  1; 
s2  =  s2  +  1; 

end 

if  s2  >  states(2) 
s2  =  1; 
si  =  si  +  1; 

end 

if  si  >  states(l) 
si  =  1; 

end 

end 


end 


end 
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7,  Define  the  Immediate  Rewards  for  each  individual  satellite 
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7,  ra(a,s)  ==>  Immediate  reward  when  choosing  action  a  while  in  state  s 
7.  (constellation) . 

7,  Assume  that  the  repair  costs  are  the  same  for  either  function  on  all 
7.  satellites 

ra  =  zeros (actions'^,  TotStates); 

for  Cons_A  =  l:actions~K 

for  Cons_S  =  1: TotStates 
for  k  =  1 :K 

if  Constellation_a{Cons_A}(k)  ==  1 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  0; 

end 

if  Constellation_a{Cons_A}(k)  ==  2 

SatState  =  Constellation_s{Cons_S}(k) ; 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  C_m{k} (SatState) ; 

end 

if  Constellation_a{Cons_A}(k)  ==  3 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  C_s(k); 

end 

end 

end 

end 


7,  Define  the  Expected  Rewards 

7,  r(s,a)  ==>  The  expected  reward  when  in  state  s  and  choosing  action  a  for 
7,  the  constellation.  This  is  equivalent  to  the  immediate  reward  plus  the 
7.  expected  penalty  cost  over  the  next  period, 
r  =  zeros (TotStates, actions^K) ; 

for  s  =  1: TotStates 

for  a  =  1: actions'^ 

7,  Calculate  the  expected  penalty  cost  over  the  next  period  -  the 
7,  sum  of  the  penalty  costs  times  the  probability  that  that  cost  is 
7,  incurred. 

Expected_penalty  =  0; 

for  j  =  1: TotStates 

Expected_penalty  =  Expected_penalty  +  Constellation_p(s ,a, j)*C_p(j) 

end 
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7.  Assign  the  expected  reward  resulting  from  taking  action  a  in 
7.  state  s . 

if  Expected_penalty  ==  0 
r(s,a)  =  NA; 

else 

r(s,a)  =  ra(a,s)  +  Expected_penalty ; 

end 


disp( [’r(5 ,int2str(s) , ’ , ’ ,int2str(a) , ,num2str(r (s,a) )] ) 


end 

end 


7.  Backward  Induction  Code 


7.  Define  the  u,  ustar  and  dstar  vector 
u  =  zeros (TotStates, actions, N) ; 
ustar  =  zeros (TotStates ,N) ; 
dstar  =  zeros (TotStates ,N) ; 

7,  Note:  The  value  of  any  action  in  the  final  time  period  has  zero 
7,  reward  and  therefore  zero  utility  ustar (s,N)  =  0  for  all  s 


7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7.7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7.7o 


n  =  N  -  1;  while  n  >=  1 

for  s  =  1: TotStates  7.  Loop  through  all  constellation  states 
ustar (s,n)  =  -realmax; 

for  a  =  l:actions~K  /  Loop  through  the  actions 

if  r(s,a)  ~=  NA  7.  Infeasible  action  -  no  reward 

expected_value  =  0; 
for  j  =  1: TotStates 

expected_value  =  expected_value  +  Constellation_p(s ,a, j ) *ustar ( j ,n+l) 
end  7.  End  For  Loop  for  expected  value 
u(s,a,n)  =  r(s,a)  +  expected_value ; 

if  u(s,a,n)  >  ustar(s,n) 
ustar(s,n)  =  u(s,a,n); 
dstar(s,n)  =  a; 

end 

end 


end 

end 
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n 


n  -  1;  7,  Decrement  the  time 


end 
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7*  Display  the  results  to  the  screen 
disp( ’Optimal  policies’); 
disp(dstar) ; 

disp(’Values  of  ustar  given  the  starting  state’); 
disp(ustar) ; 

7,  Open  file  in  current  directory  to  write  results  to 
fid  =  f open ( ’Three_Sat_Solut ion. txt ’, ’w’ ) ; 

7,  Write  the  results  to  the  file 
fprintf(fid,  ’Backward  Induction  run  time  =  ’); 
fprintf (fid,  ’7«8.5f\t’,  BackwardlnductionRunTime)  ; 
fprintf (fid,  ’\n  Optimal  Policy’);  fprintf (fid,  ’\n’);  for  i  = 
1 :TotStates ; 
for  j=l : N 

fprintf  (fid,  ’7«g\t’  ,dstar(i,  j)) ; 

end 

fprintf (fid,  ’\n’); 

end  fprintf (fid,  ’\n  Optimal  Values  \n’);  for  i  =  l:TotStates; 
for  j=l : N 

fprintf  (fid,  ’7»8 . 5f  \t  ’  ,  ustar  (i ,  j  )  )  ; 

end 

fprintf (fid,  ’\n’); 

end 

7,  Close  file 
f close(f id) ; 


B-16 


Appendix  C.  One-Satellite  One-Way  Analysis 


%7.7oy.y.%7.%yo%%7.%yo%%7//.7o%y//.7.7o7.%%7.7oyo%%7.y.y.7.%%0/.7o%%%0/.7o7.0/o%7.7oyoy.%%7.0/.y.7.%%%7o7.%%7.7oyo%%%7.7o7.% 

7,  Author:  lLt  Tim  Cook 
7.  AFIT/ENS/G0R-05M 


7.  March  2005 

7,  This  program  performs  a  one-way  sensitivity  analysis  on  a  Markov 
7,  decision  process  formulated  to  find  an  optimal  maintenance  policy  for  a 
7,  one  satellite  constellation.  In  this  file,  the  parameters  that  are 
7,  modified  are  the  space  vehicle  cost  and  the  replacement  cost. 


7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7«7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7.7o 


7,  Begin  the  outer  for  loop  that  performs  the  sensitivity  analysis.  The 
7,  first  time  through,  the  launch  vehicle  costs  are  varied  while  the 
7,  replacement  cost  is  held  constant  at  $500  million.  The  second  time 
7,  through,  the  replacement  cost  is  varied  while  the  space  vehicle  cost  is 
7.  held  constant  at  $450  million, 
for  parameter  =  1:2 

7.  Define  Parameters 
N  =  20;  7.  Length  of  planning  horizon 
M  =  3;  7#  Number  of  functions 

7.  Total  number  of  states  that  the  satellite  can  possibly  enter  into 
TotStates  =  2~M; 

7.  Number  of  possible  actions 
actions  =  3; 

7.  Failure  rates  —  1/ (expected  lifetime  in  years)* (4  quarters) 
lambdal  =  1/(5. 5*4);  7«  Failure  rate  of  function  1 

lambda2  =  1/(5. 25*4);  7«  Failure  rate  of  function  2 

lambda3  =  1/(6. 5*4);  °/  Failure  rate  of  function  3 

7.  Penalty  costs 

C_p  =  [0  -200  -500  -600  -300  -400  -400  -700] ; 

7.  Allocate  memory  for  the  matrices  that  will  store  the  sensitivity 
l  results  before  they  are  output  to  a  file 
results  =  zeros (TotStates,  100); 
policies  =  cell(lOO); 
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for  base  =  1 : 100 
disp(base) 

7,  On  orbit  repair  cost  by  state 
if  parameter  ==  1 

Launch  =  base* 10; 

else 

Launch  =  450; 

end 

C_m  =  [-realmax  -(Launch  +  20)  -(Launch  +  15)  -(Launch  +  35)  -(Launch  +  35)  -(Launch  +  55) 
-(Launch  +  50)  -(Launch  +  70)] ; 

7,  Satellite  replacement  cost 
if  parameter  ==  2 
C_s  =  -base*10; 

else 

C_s  =  -500; 

end 

7,  Use  when  an  action  is  not  feasible  for  that  state 
NA  =  -99999; 

7,  Failure  probabilities  for  each  function 
F  =  [1-exp (-lambdal)  1-exp (-lambda2)  1-exp (-lambda3) ] ; 

7,  Survival  probabilities  for  each  function 
R  =  [exp (-lambdal)  exp(-lambda2)  exp(-lambda3)] ; 

7.  Probability  of  a  successful  repair  for  each  function 
M  =  [.95  .96  .97] ; 

7,  Probability  of  a  successful  replacement  for  each  function 
G  =  [.975  .94  .98] ; 


7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7o7o7o7.7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o 


7,  Define  Transition  Probabilities 

7,  p(s,a,j)  ==>  Probability  of  going  to  state  j,  given  presently  in 
7,  state  s  and  choosing  action  a 

p  =  zeros (TotStates ,  actions,  TotStates) ; 


p(l,l,l) 

p(l , 1,2) 
p(l , 1 ,3) 
p(l , 1 ,4) 
p(l , 1 ,5) 
p(l , 1 , 6) 
p(l , 1 ,7) 
p(l , 1 , 8) 


=  R(1)*R(2)*R(3) ; 
=  R(l) *R(2) *F (3) ; 
=  R(l) *F (2) *R(3) ; 
=  R(l) *F (2) *F (3) ; 
=  F(l) *R(2) *R(3) ; 
=  F(l) *R(2) *F (3) ; 
=  F(l) *F (2) *R(3) ; 
=  F(l) *F (2) *F (3) ; 
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7.  There  is  no  need  to  include  the  transition  probabilities  for  action  2 
"/,  while  in  state  2  because  the  action  will  never  be  chosen.  This  is  because 
7.  the  transition  probabilities  are  identical  to  those  of  action  1,  but  the 


7,  large 

p(l ,3, 1) 
p(l ,3, 2) 
p(l ,3,3) 
p(l ,3,4) 
p(l ,3,5) 
p(l,3,6) 
p(l ,3,7) 
p(l ,3,8) 

p(2 , 1 ,2) 

p(2 , 1 ,4) 
p(2,l,6) 
p(2 , 1 ,8) 

p(2 ,2 , 1) 
p(2,2,2) 

p(2,2,3) 

p(2,2,4) 

p(2,2,5) 

p(2,2,6) 

p(2,2,7) 

p(2,2,8) 

p(2 ,3, 1) 
p(2,3,2) 
p(2,3,3) 
p(2,3,4) 
p(2,3,5) 
p(2,3,6) 
p(2,3,7) 
p(2,3,8) 

p(3 , 1 ,3) 
p(3 , 1 ,4) 
p(3 , 1 ,7) 
p(3 , 1 ,8) 


setup  cost  is  not  incurred  with  action  1. 


=  G(1)*G(2)*G(3) ; 

=  G(1)*G(2)*(1-G(3)) ; 

=  G(1)*(1-G(2))*G(3) ; 

=  G(l) *(1-G(2) ) * (1-G(3) ) ; 

=  (1-G(1))*G(2)*G(3) ; 

=  (l-G(l) ) *G(2) * (1-G(3) ) ; 

=  (l-G(l) ) * (1-G(2) ) *G(3) ; 

=  ( 1— G ( 1 ) ) * ( 1— G (2) ) * ( 1— G (3) ) ; 

=  R(1)*R(2); 

=  R(1)*F(2) ; 

=  F(1)*R(2); 

=  F(1)*F(2); 

=  R(1)*R(2)*M(3) ; 

=  R(1)*R(2)*(1-M(3)) ; 

=  R(1)*F(2)*M(3) ; 

=  R(1)*F(2)*(1-M(3)) ; 

=  F(1)*R(2)*M(3) ; 

=  F(1)*R(2)*(1-M(3)) ; 

=  F(1)*F(2)*M(3) ; 

=  F(1)*F(2)*(1-M(3)) ; 

=  G(1)*G(2)*G(3) ; 

=  G(1)*G(2)*(1-G(3)) ; 

=  G(1)*(1-G(2))*G(3) ; 

=  G(l) *(1-G(2) ) * (1-G(3) ) ; 

=  (1-G(1))*G(2)*G(3) ; 

=  (l-G(l) ) *G(2) * (1-G(3) ) ; 

=  (l-G(l) ) * (1-G(2) ) *G(3) ; 

=  ( 1— G ( 1 ) ) * ( 1— G (2) ) * ( 1— G (3) ) ; 

=  R(1)*R(3) ; 

=  R(1)*F(3) ; 

=  F(1)*R(3) ; 

=  F(1)*F(3); 
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p(3 ,2 , 1) 
p(3,2,2) 
p(3,2,3) 
p(3,2,4) 
p(3,2,5) 
p(3,2,6) 
p(3,2,7) 
p(3,2,8) 

p(3,3,l) 

p(3,3,2) 

p(3,3,3) 

p(3,3,4) 

p(3,3,5) 

p(3,3,6) 

p(3,3,7) 

p(3,3,8) 

p(4,l,4) 
p(4, 1 ,8) 

p(4,2,l) 

p(4,2,2) 

p(4,2,3) 

p(4,2,4) 

p(4,2,5) 

p(4,2,6) 

p(4,2,7) 

p(4,2,8) 

p(4,3,l) 

p(4,3,2) 

p(4,3,3) 

p(4,3,4) 

p(4,3,5) 

p(4,3,6) 

p(4,3,7) 

p(4,3,8) 

p(5,l,5) 
p(5,l,6) 
p(5 , 1 ,7) 
p(5,l,8) 


R(1)*M(2)*R(3) ; 
R(1)*M(2)*F(3) ; 
R(1)*(1-M(2))*R(3) ; 
R(1)*(1-M(2))*F(3)  ; 
F(1)*M(2)*R(3) ; 
F(1)*M(2)*F(3) ; 
F(1)*(1-M(2))*R(3)  ; 
F(1)*(1-M(2))*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3))  ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 

( 1— G ( 1 ) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 


R(l); 
F(l)  ; 


R(1)*M(2)*M(3) ; 
R(1)*M(2)*(1-M(3)) ; 
R(1)*(1-M(2))*M(3) ; 
R(l)*(l-M(2))*(l-M(3)) ; 
F(1)*M(2)*M(3) ; 
F(1)*M(2)*(1-M(3)) ; 
F(1)*(1-M(2))*M(3) ; 
F(l)*(l-M(2))*(l-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(l-G(l) ) * (1-G(2) ) *(1-G(3) ) ; 

R(2)*R(3) ; 

R(2)*F(3) ; 

F(2)*R(3) ; 

F(2)*F(3) ; 
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p(5 ,2 , 1) 
p(5,2,2) 
p(5,2,3) 
p(5,2,4) 
p(5,2,5) 
p(5,2,6) 
p(5,2,7) 
p(5,2,8) 

p(5,3,l) 

p(5,3,2) 

p(5,3,3) 

p(5,3,4) 

p(5,3,5) 

p(5,3,6) 

p(5,3,7) 

p(5,3,8) 

p(6 , 1 ,6) 

p(6,l,8) 

p(6,2,l) 

p(6,2,2) 

p(6,2,3) 

p(6,2,4) 

p(6,2,5) 

p(6,2,6) 

p(6,2,7) 

p(6,2,8) 

p(6,3,l) 

p(6,3,2) 

p(6,3,3) 

p(6,3,4) 

p(6,3,5) 

p(6,3,6) 

p(6,3,7) 

p(6,3,8) 

p(7, 1 ,7) 
p(7,l,8) 


M(1)*R(2)*R(3) ; 
M(1)*R(2)*F(3) ; 
M(1)*F(2)*R(3) ; 
M(1)*F(2)*F(3) ; 
(1-M(1))*R(2)*R(3) ; 
(1-M(1))*R(2)*F(3) ; 
(1-M(1))*F(2)*R(3) ; 
(1-M(1))*F(2)*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

( 1— G ( 1 ) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 

(1-G (!))*(! -G(2))*(l-G (3)) ; 


R(2)  ; 
F (2)  ; 


M(1)*R(2)*M(3) ; 
M(1)*R(2)*(1-M(3)) ; 
M(1)*F(2)*M(3) ; 
M(1)*F(2)*(1-M(3)) ; 
(1-M(1))*R(2)*M(3) ; 
(1-M(1))*R(2)*(1-M(3)) ; 
(1-M(1))*F(2)*M(3) ; 

(l-M(l) ) *F(2) * (1-M(3) ) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(l-G(l) ) *G(2) *G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 


R(3)  ; 
F (3)  ; 
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p(7, 

to 

p(7, 

2,2) 

P(7, 

2,3) 

P(7, 

2,4) 

P(7, 

2,5) 

P(7, 

2,6) 

P(7, 

2,7) 

P(7, 

2,8) 

P(7, 

3,1) 

P(7, 

3,2) 

P(7, 

3,3) 

P(7, 

3,4) 

P(7, 

3,5) 

P(7, 

3,6) 

P(7, 

3,7) 

P(7, 

3,8) 

00 

Oh 

1,8) 

T3 

00 

2,1) 

00 

Oh 

2,2) 

T3 

00 

2,3) 

00 

Oh 

2,4) 

T3 

00 

2,5) 

T3 

00 

2,6) 

00 

Oh 

2,7) 

T3 

00 

2,8) 

T3 

00 

3,1) 

00 

Oh 

3,2) 

00 

3,3) 

T3 

00 

3,4) 

00 

Oh 

3,5) 

T3 

00 

3,6) 

00 

Oh 

3,7) 

00 

3,8) 

M(1)*M(2)*R(3) ; 
M(1)*M(2)*F(3) ; 
M(1)*(1-M(2))*R(3) ; 
M(1)*(1-M(2))*F(3) ; 
(1-M(1))*M(2)*R(3) ; 
(1-M(1))*M(2)*F(3) ; 
(1-M(1))*(1-M(2))*R(3) ; 
(1-M(1))*(1-M(2))*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(l-G(l) ) * (1-G(2) ) *(1-G(3) ) ; 

1; 

M(1)*M(2)*M(3) ; 
M(1)*M(2)*(1-M(3)) ; 
M(1)*(1-M(2))*M(3) ; 
M(l)*(l-M(2))*(l-M(3)) ; 
(1-M(1))*M(2)*M(3) ; 

(l-M(l) ) *M(2) * (1-M(3) ) ; 
(1-M(1))*(1-M(2))*M(3) ; 
(l-M(l) ) * (1-M(2) ) * (1-M(3) ) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


7o7oy////.7oyoy//o7.7oyo%7o7.0///o7o%7.%7o7.%7.%7o7.%7.7.7oyo%7//.0/.7.7.7//.7oyo7.%7//o7o7o%%%7.7oyo%%7.7.7o7.%7//.7o7o% 


"/,  Define  the  Immediate  Rewards  -  deterministic  part  of  the  reward 
*/,  function  ra{a}(s)  ==>  Immediate  reward  when  choosing  action  a  in 
7,  state  s 
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ra  =  cell (actions) ; 


ra{l}  =  zeros (TotStates) ; 
ra{2}  =  zeros (TotStates) ; 
ra{3}  =  zeros (TotStates) ; 


ra{2}(l)  =  NA; 
ra{2}(2)  =  C_m(2) 
ra{2>(3)  =  C_m(3) 
ra{2}(4)  =  C_m(4) 
ra{2>(5)  =  C_m(5) 
ra{2>(6)  =  C_m(6) 
ra{2}(7)  =  C_m(7) 
ra{2>(8)  =  C_m(8) 

ra{3}(l)  =  C_s; 
ra{3}(2)  =  C_s; 
ra{3>(3)  =  C_s; 
ra{3}(4)  =  C_s; 
ra{3}(5)  =  C_s; 
ra{3>(6)  =  C_s; 
ra{3}(7)  =  C_s; 
ra{3}(8)  =  C_s; 


l  Define  the  Expected  Rewards 

/  r(s,a)  ==>  The  expected  reward  when  in  state  s  and  choosing 
°/0  action  a.  This  is  equivalent  to  the  immediate  reward  plus  the 
°/0  expected  penalty  cost  over  the  next  period, 
r  =  zeros (TotStates , actions) ; 


for  s  =  1: TotStates 
for  a  =  1: actions 

°/0  Calculate  the  expected  penalty  cost  over  the  next  period 
'/.  (the  sum  of  the  penalty  costs  times  the  probability  that 
7,  that  cost  is  incurred)  . 

Expected_Penalty  =  0; 

for  j  =  1: TotStates 

Expected_Penalty  =  Expected_Penalty  +  p(s,a, j)*C_p(j) ; 

end 

7,  Assign  the  expected  reward  resulting  from  taking  action  a 
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7.  in  state  s . 
if  Expected_Penalty  ==  0 

r(s,a)  =  NA;  °/0  The  action  is  not  feasible 

else 

r(s,a)  =  ra{a}(s)  +  Expected_Penalty ; 

end 

end 

end 


rararammmrararammmmrararamraramnran 


/  Backward  Induction  Code:  Directly  adapted  from  Capt  Sumter’s  code 


%  Define  the  u,  ustar  and  dstar  vector 
u  =  zeros (TotStates , actions ,N) ; 
ustar  =  zeros (TotStates ,N) ; 
dstar  =  zeros (TotStates ,N) ; 

%  Note:  The  value  of  any  action  in  the  final  time  period  has  zero 
7.  reward  and  therefore  zero  utility 
7,  ustar (s,N)  =  0  for  all  s 


tic  7o  Starts  the  timer  used  to  measure  the  time  to  solve  the  MDP 


n  =  N  -  1; 
while  n  >=  1 

for  s  =  1: TotStates 

ustar (s,n)  =  -realmax; 
for  a  =  1: actions 

if  r(s,a)  ~=  NA  7*  Infeasible  action  -  no  reward 
expected_value  =  0; 
for  j  =  l:TotStates 

expected_value  =  expected_value  +  p(s,a, j)*ustar(j ,n+l) 

end 

u(s,a,n)  =  r(s,a)  +  expected_value ; 


if  u(s,a,n)  >  ustar(s,n) 
ustar(s,n)  =  u(s,a,n); 
dstar(s,n)  =  a; 

end 

end 

end 

end 

n  =  n  -  1;  7«  Decrement  the  time 

end 
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toe  #/«  Stops  the  timer  used  to  measure  the  time  to  solve  the  MDP 


ramrammmrararay.mrararay.mrararararararan 


7.  Store  the  optimal  value  (ustar)  results  in  the  matrix  ’results’ 
results (1,  base)  =  base; 
for  index  =  l:TotStates 

results (index,  base)  =  -ust ar( index, 1) ; 


end 


/  Store  the  optimal  policies  (dstar)  in  the  cell  array  "policies" 
policies{base}  =  dstar; 

end  7,  end  the  loop  that  controls  the  sensitivity  analysis 

7.  Determine  which  file  to  write  to 
if  parameter  ==  1 

fid  =  f open( ’One_Sat_Repair_Cost . txt ’ , ’w’ ) ; 

else 

fid  =  f open( ’One_Sat_Replace_Cost .txt ’ , ’w’ ) ; 

end 


7.  Write  the  results  to  a  file  in  the  current  directory 
for  j=l:base 

fprintf(fid,  ’7o8.5f\t’,  j); 

end 

fprintf(fid,  ’\n’); 
for  i  =  l:TotStates; 
for  j=l:base 

fprintf  (fid,  ’7«8.5f\t ’  , results (i ,  j ) )  ; 

end 

fprintf (fid,  ’\n’); 

end 


for  j=l:base 

fprintf (fid,  ’\n  Parameter  Iteration  -  ’); 
fprintf  (fid,  ’7»g\t’,  j); 
fprintf (fid,  ’\n’); 
for  s=l :TotStates 
for  n=l:N 

fprintf  (fid,  ’7og\t’,  policies{j}  (s  ,n) ) ; 

end 

fprintf (fid,  ’\n’); 

end 


C-9 


end 


l  Close  file 
f close(f id) ; 

end 
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Appendix  D.  Three-Satellite  One-Way  Analysis 


%7.7o7.y.%7.%%%%7.%yo%%7//.7o%%%7.7o7.%%7.7oyo%%7.%y.7.%%7.7o7.%%0/.7o7.0/o%7.7oyoy.%%7.0/.y.7.%%%7o7.%%7.7o7o%%%7.7o7.% 

7,  Author:  lLt  Tim  Cook 
7.  AFIT/ENS/G0R-05M 


7.  March  2005 

7,  This  program  performs  a  one-way  sensitivity  analysis  on  a  Markov 
7,  decision  process  formulated  to  find  an  optimal  maintenance  policy  for  a 
7,  three  satellite  constellation.  In  this  file,  the  parameters  that  are 
7,  modified  are  the  space  vehicle  cost  and  the  replacement  cost. 


7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7«7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7.7o 


7,  Define  Parameters 

N  =  20;  7o  Length  of  planning  horizon 

M  =  [2  2  3]  ;7»  Number  of  functions  desired  operational  in  each  satellite 

K  =  3;  7.  Number  of  Satellites 

7,  Create  a  vector  that  keeps  the  number  of  degradation  levels  for  each 

7,  satellite 

states  =  zeros (K);  for  i=l:K 

states(i)  =  2~M(i) ;  °/  Number  of  possible  states  for  satellite  i 

end 


7,  Define  the  total  number  of  states  that  the  entire  satellite 
7.  constellation  can  possibly  enter  into 
TotStates  =  prod(states ( : , 1) ) ; 


7,  Number  of  possible  actions  for  each  satellite 
actions  =  3; 


7,  Failure 
lambdal  = 
lambda2  = 
lambda3  = 


rates  —  1/ (expected  lifetime  in  years)* (4  quarters) 
1/(5. 5*4);  7o  Failure  rate  of  function  1 

1/(5. 25*4);  7o  Failure  rate  of  function  2 

1/(6. 5*4);  7*  Failure  rate  of  function  3 


7.  Allocate  memory  for  the  matrices  that  will  store  the  sensitivity 

7,  results  before  they  are  output  to  a  file 

results  =  zeros (TotStates,  100);  policies  =  cell(100); 


7.  Use  when  an  action  is  not  feasible  for  that  state 

NA  =  -99999; 
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7,  Failure  probabilities  for  each  function 
F  =  [1-exp (-lambdal)  1-exp (-lambda2)  1-exp (-lambda3)] ; 

%  Survival  probabilities  for  each  function 
R  =  [exp (-lambdal)  exp(-lambda2)  exp(-lambda3)] ; 

7*  Probability  of  a  successful  repair  for  each  function 
M  =  [.95  .96  .97] ; 

7,  Probability  of  a  successful  replacement  for  each  function 
G  =  [.975  .94  .98] ; 


7.7.7o7o7o7o7.7o7o7o7o7.7o7o7o7o7.7X/o7o7.7.7o7o7o7.7o7o7o7o7o7.7o7o7o7o7.7o7o7o7o7.7.7,7o7o7o7.7o7o7o7o7o7.7o7o7o7o7o7o7o7o7o7.7.7o7o7o7o7. 


7,  Define  Transition  Probabilities 

7,  p{kXs,a,j)  ==>  Probability  for  satellite  k  of  going  to  state  j, 

7,  given  presently  in  state  s  and  choosing  action  a  for 

7,  that  individual  satellite 


7,  Allocate  memory  for  the  transition  probability  matricies 
7,  using  cell  arrays 
p  =  cell(K);  for  i  =  1:K 

p(i)  =  {zeros(states(i) ,  actions,  states(i))}; 

end 


7,  Define  the  transition  probabilities 
p{l}(l, 1, 1)  =  R(1)*R(2); 
p{l}(l,l,2)  =  R(l) *F (2) ; 
p{l}(l,l,3)  =  F(1)*R(2); 

P{1}(1, 1,4)  =  F(l) *F (2) ; 

7.  There  is  no  need  to  include  the  transition  probabilities  for  action  2 
7,  while  in  state  2  because  the  action  will  never  be  chosen.  This  is  because 
7,  the  transition  probabilities  are  identical  to  those  of  action  1,  but  the 
7,  large  setup  cost  is  not  incurred  with  action  1.  The  same  holds  for  the  other 
7.  two  satellites. 


p{l} (1 , 3 , 1) 
p{l}(l,3,2) 
p{l}(l,3,3) 
p{l}(l,3,4) 


G(l) *G(2) ; 

G(l) * ( 1— G (2) ) ; 
(l-G(l) ) *G(2) ; 
(l-G(l) ) * (1-G(2) ) ; 


p{l}(2,l,2)  =  R(l) ; 
p{l}(2, 1,4)  =  F(l) ; 


p{l>(2,2,l)  =  R(l) *M(2) ; 
p{l}(2,2,2)  =  R(l) * (1-M(2) ) ; 
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p{l}(2,2,3)  =  F(1)*M(2); 
p{l}(2,2,4)  =  F(l)*(l-M(2)); 


p{l}(2,3,l)  =  G(1)*G(2); 
p{l}(2,3,2)  =  G(l)*(l-G(2)); 
p{l>(2,3,3)  =  (l-G(i))*G(2); 
p{l>(2,3,4)  =  (1-G(1))*(1-G(2)); 

p{l} (3 , 1 , 3)  =  R(2) ; 
p{l} (3,1,4)  =  F (2) ; 

p{l} (3 , 2 , 1)  =  M(1)*R(2); 
p{l>(3,2,2)  =  M(1)*F(2); 
p{l}(3,2,3)  =  (1-M(1))*R(2); 
p{l} (3,2,4)  =  (1-M(1))*F(2); 

p{l>(3,3,l)  =  G(1)*G(2); 
p{l}(3,3,2)  =  G(l)*(l-G(2)); 
p{l}(3,3,3)  =  (1-G(1))*G(2); 
p{l} (3,3,4)  =  (1-G(l))*(i-G(2)); 

p{l} (4,1,4)  =  1; 

p{l}(4,2, 1)  =  M(1)*M(2); 
p{l>(4,2,2)  =  M(l)*(l-M(2)) ; 
p{l>(4,2,3)  =  (l-M(l) ) *M(2) ; 
p{l>(4,2,4)  =  (1-M(1))*(1-M(2)) ; 

p{l}(4,3, 1)  =  G(1)*G(2) ; 
p{l>(4,3,2)  =  G(l)*(l-G(2)); 
p{l>(4,3,3)  =  (1-G(1))*G(2); 
p{l}(4,3,4)  =  (1-G(1))*(1-G(2)); 
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p{2>(l,l,l)  =  R(l) *R(2) ; 
p{2>(l,l,2)  =  R(1)*F(2) ; 
p{2}(l,l,3)  =  F(1)*R(2); 
p{2} (1 ,1,4)  =  F(1)*F(2); 


p{2} (1 ,3,1)  =  G(1)*G(2); 
p{2}(l,3,2)  =  G(l)*(l-G(2)) ; 
p{2}(l,3,3)  =  (1-G(1))*G(2) ; 


p{2}(l,3,4) 


(1-G(1))*(1-G(2)); 


p{2} (2,1,2)  =  R(l) ; 
p{2} (2,1,4)  =  F(l) ; 


p{2} (2,2,1) 
p{2}(2,2,2) 

p{2}(2,2,3) 

p{2}(2,2,4) 


R(l) *M(2) ; 

R(l) *(1-M(2) ) ; 
F(l) *M(2) ; 

F(l) *(1-M(2) ) ; 


p{2>(2>3,l) 

p{2}(2,3,2) 

p{2}(2,3,3) 

p{2}(2,3,4) 


G(l) *G(2) ; 
G(l)*(l-G(2)) ; 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)); 


p{2}(3,l,3)  =  R(2) ; 
p{2} (3,1)4)  =  F (2) ; 


p{2} (3,2,1) 
p{2}(3,2,2) 
p{2}(3,2,3) 
p{2} (3,2,4) 


M(l) *R(2) ; 

M(l) *F (2) ; 
(1-M(1))*R(2) ; 
(1-M(1))*F(2) ; 


p{2} (3,3,1) 
p{2}(3,3,2) 
p{2}(3,3,3) 
p{2} (3,3,4) 


G(l) *G(2) ; 

G(l) *(1-G(2) ) ; 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)); 


p{2} (4,1)4)  =  1; 


p{2} (4,2,1) 
p{2}(4,2,2) 
p{2}(4,2,3) 
p{2}(4,2,4) 


M(1)*M(2) ; 
M(l)*(l-M(2))  ; 
(1-M(1))*M(2) ; 
(1-M(1))*(1-M(2)) ; 


p{2} (4,3,1) 
p{2}(4,3,2) 
p{2}(4,3,3) 
p{2}(4,3,4) 


G(l) *G(2) ; 
G(l)*(l-G(2)) ; 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)); 
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p{3>(l,l,l)  =  R(1)*R(2)*R(3) ;  p{3}(l,l,2)  =  R(l) *R(2) *F(3) ; 
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p{3} (1 ,1,3) 
p{3} (1 ,1,5) 
p{3} (1 ,1,7) 

p{3}(l,3,l) 
p{3>(l,3,2) 
p{3}(l,3,3) 
p{3} (1 ,3,4) 
p{3}(l,3,5) 
p{3>(l,3,6) 
p{3}(l,3,T) 
p{3Xl,3,8) 

p{3} (2,1,2) 
p{3} (2,1,4) 
p{3X2>l,6) 
p{3X2>l,8) 

p{3X2,2,l) 
p{3}(2,2,2) 
p{3}(2,2,3) 
p{3} (2,2,4) 
p{3} (2,2,5) 
p{3}(2,2,6) 
p{3}(2,2,7) 
p{3}(2,2,8) 

p{3X2,3,l) 
p{3}(2,3,2) 
p{3}(2,3,3) 
p{3} (2,3,4) 
p{3}(2,3,5) 
p{3}(2,3,6) 
p{3}(2,3,7) 
p{3}(2,3,8) 

p{3X3, 1  >3) 
p{3X3,l,4) 
p{3X3,l,7) 
p{3X3,l,8) 

p{3} (3,2,1) 
p{3}(3,2,2) 


R(l) *F (2) *R(3) ;  p{3}(l,l,4)  =  R(l) *F(2) *F(3) 
F(l)  *R(2)  *R(3)  ;  p{3Xl,l,6)  =  F(l)  *R(2)  *F(3) 
F(l) *F (2) *R(3) ;  p{3}(l,l,8)  =  F(l) *F(2) *F(3) 

G(l) *G(2) *G(3) ; 

G(1)*G(2)*(1-G(3)) ; 

G(1)*(1-G(2))*G(3) ; 

G(l)*(l-G(2))*(l-G(3)) ; 

(1-G(1))*G(2)*G(3) ; 

(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

R(l) *R(2) ; 

R(1)*F(2) ; 

F(l) *R(2) ; 

F(l) *F (2) ; 

R(l) *R(2) *M(3) ; 

R(1)*R(2)*(1-M(3)) ; 

R(l) *F (2) *M(3) ; 

R(l) *F(2) *(1-M(3) ) ; 

F(l) *R(2) *M(3) ; 

F(1)*R(2)*(1-M(3)) ; 

F(l) *F (2) *M(3) ; 

F(1)*F(2)*(1-M(3)) ; 

G(l) *G(2) *G(3) ; 

G(1)*G(2)*(1-G(3)) ; 

G(1)*(1-G(2))*G(3) ; 

G(l)*(l-G(2))*(l-G(3)) ; 

(1-G(1))*G(2)*G(3) ; 

(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

R(l) *R(3) ; 

R(1)*F(3) ; 

F(l) *R(3) ; 

F(1)*F(3) ; 

R(l) *M(2) *R(3) ; 

R(l) *M(2) *F (3) ; 
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p{3}(3,2,3) 

p{3}(3,2,4) 

p{3}(3,2,5) 

p{3}(3,2,6) 

p{3}(3,2,7) 

p{3}(3,2,8) 

p{3} (3,3,1) 
p{3}(3,3,2) 
p{3}(3,3,3) 
p{3}(3,3,4) 
p{3}(3,3,5) 
p{3}(3,3,6) 
p{3}(3,3,7) 
p{3}(3,3,8) 

p{3} (4,1,4) 
p{3} (4,1,8) 

p{3} (4,2,1) 
p{3}(4,2,2) 
p{3}(4,2,3) 
p{3}(4,2,4) 
p{3}(4,2,5) 
p{3}(4,2,6) 
p{3}(4,2,7) 
p{3}(4,2,8) 

p{3}(4,3, 1) 
p{3}(4,3,2) 
p{3}(4,3,3) 
p{3}(4,3,4) 
p{3}(4,3,5) 
p{3}(4,3,6) 
p{3}(4,3,7) 
p{3}(4,3,8) 

p{3} (5,1,5) 
p{3} (5,1,6) 
p{3} (5,1,7) 
p{3} (5,1,8) 

p{3}(5,2, 1) 


R(1)*(1-M(2))*R(3) ; 
R(1)*(1-M(2))*F(3) ; 

F(l) *M(2) *R(3) ; 

F(l) *M(2) *F (3) ; 
F(1)*(1-M(2))*R(3) ; 
F(1)*(1-M(2))*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(l) ; 
F(l) ; 


R(l) *M(2) *M(3) ; 

R(l) *M(2) *(1-M(3) ) ; 
R(1)*(1-M(2))*M(3) ; 
R(l)*(l-M(2))*(l-M(3)) ; 
F(1)*M(2)*M(3) ; 
F(1)*M(2)*(1-M(3)) ; 
F(1)*(1-M(2))*M(3) ; 
F(l)*(l-M(2))*(l-M(3)) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

R(2) *R(3) ; 

R(2) *F (3) ; 

F (2) *R(3) ; 

F (2) *F (3) ; 

M(l) *R(2) *R(3)  ; 
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p{3}(5,2,2) 

p{3}(5,2,3) 

p{3}(5,2,4) 

p{3}(5,2,5) 

p{3}(5,2,6) 

p{3}(5,2,7) 

p{3}(5,2,8) 

p{3} (5,3,1) 
p{3}(5,3,2) 
p{3}(5,3,3) 
p{3}(5,3,4) 
p{3}(5,3,5) 
p{3}(5,3,6) 
p{3}(5,3,7) 
p{3}(5,3,8) 

p{3} (6,1,6) 
p{3} (6,1,8) 

p{3}(6,2,l) 
p{3}(6,2,2) 
p{3} (6,2,3) 
p{3}(6,2,4) 
p{3}(6,2,5) 
p{3}(6,2,6) 
p{3} (6,2,7) 
p{3} (6,2,8) 

p{3}(6,3,l) 
p{3}(6,3,2) 
p{3}(6,3,3) 
p{3} (6,3,4) 
p{3}(6,3,5) 
p{3}(6,3,6) 
p{3}(6,3,7) 
p{3}(6,3,8) 

p{3} (7,1,7) 
p{3} (7,1,8) 

p{3} (7,2,1) 
p{3}(7,2,2) 


M(l) *R(2) *F (3) ; 
M(1)*F(2)*R(3) ; 

M(l) *F (2) *F (3) ; 
(1-M(1))*R(2)*R(3) ; 
(1-M(1))*R(2)*F(3) ; 
(1-M(1))*F(2)*R(3) ; 
(1-M(1))*F(2)*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)) ; 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(2); 
F (2)  ; 


M(1)*R(2)*M(3) ; 
M(1)*R(2)*(1-M(3)) ; 
M(1)*F(2)*M(3) ; 
M(1)*F(2)*(1-M(3)) ; 
(1-M(1))*R(2)*M(3) ; 
(1-M(1))*R(2)*(1-M(3)) ; 
(1-M(1))*F(2)*M(3) ; 
(1-M(1))*F(2)*(1-M(3)) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(3); 
F(3)  ; 


M(1)*M(2)*R(3) ; 
M(1)*M(2)*F(3) ; 
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p{3}(7,2,3) 
p{3}(7,2,4) 
p{3}(7 ,2,5) 
p{3}(7,2,6) 
p{3}(7,2,7) 
p{3}(7 ,2,8) 


M(1)*(1-M(2))*R(3) ; 
M(1)*(1-M(2))*F(3) ; 
(1-M(1))*M(2)*R(3) ; 
(1-M(1))*M(2)*F(3) ; 
(1-M(1))*(1-M(2))*R(3) ; 
(1-M(1))*(1-M(2))*F(3) ; 


p{3} (7,3,1) 
p{3}(7,3,2) 
p{3}(7,3,3) 
p{3}(7,3,4) 
p{3}(7,3,5) 
p{3}(7,3,6) 
p{3}(7 ,3,7) 
p{3}(7,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


p{3} (8,1,8)  =  1; 


p{3} (8,2,1) 
p{3}(8,2,2) 
p{3}(8,2,3) 
p{3} (8,2,4) 
p{3} (8,2,5) 
p{3}(8,2,6) 
p{3}(8,2,7) 
p{3}(8,2,8) 


M(1)*M(2)*M(3)  ; 
M(1)*M(2)*(1-M(3)) ; 
M(1)*(1-M(2))*M(3) ; 
M(l)*(l-M(2))*(l-M(3)) ; 
(1-M(1))*M(2)*M(3) ; 

(l-M(l) ) *M(2) * (1-M(3) ) ; 
(1-M(1))*(1-M(2))*M(3) ; 
(l-M(l) ) * (1-M(2) ) *(1-M(3) ) ; 


p{3}(8,3, 1) 
p{3}(8,3,2) 
p{3}(8,3,3) 
p{3} (8,3,4) 
p{3}(8,3,5) 
p{3}(8,3,6) 
p{3}(8,3,7) 
p{3}(8,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3))  ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3); 
(1-G(1))*(1-G(2))*(1-G(3)); 
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"/,  Define  the  state  space  for  each  satellite 
Sat_s  =  cell(K);  Sat_s{l}  =[11 


1  0 
0  1 


0  0]  ; 

Sat_s{2>  =[11 
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1  0 
0  1 
0  0]  ; 

Sat_s{3>  =[111 
110 
10  1 
10  0 
Oil 
0  10 
0  0  1 
0  0  0]; 


l  Define  the  overall  state  space  for  the  constellation 
Constellation_s  =  cell(TotStates) ; 
si  =  1; 
s2  =  1; 
s3  =  1 ; 

for  i  =  l:TotStates 

Constellation_s{i}  =  [si  s2  s3] ; 


s3  =  s3  +  1; 

if  s3  >  states (3) 
s3  =  1; 
s2  =  s2  +  1; 

end 

if  s2  >  states (2) 
s2  =  1; 
si  =  si  +  1; 

end 

if  si  >  states(l) 
si  =  1; 

end 


end 


7,  Penalty  Cost  assigned  per  state 
7,  The  decision  maker  desires  that 
C_p  =  zeros (TotStates) ;  for  i  =  l:TotStates 

’/,  Function  1  Penalty  Cost 
si  =  Constellation_s{i}(l) ; 
s2  =  Constellation_s{i}(2) ; 
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s3  =  Constellation_s{i}(3) ; 

sum  =  Sat_s{l}(sl , 1)+Sat_s{2}(s2, 1)+Sat_s{3}(s3, 1) ; 
if  sum  ==  3 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  2 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  1 

C_p(i)  =  C_p(i)  -  500; 
elseif  sum  ==  0 

C_p(i)  =  C_p(i)  -  650; 

end 

l  Function  2  Penalty  Cost 

sum  =  Sat_s{l}(sl ,2)+Sat_s{2}(s2,2)+Sat_s{3}(s3,2) ; 
if  sum  ==  3 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  2 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  1 

C_p(i)  =  C_p(i)  -  450; 
elseif  sum  ==  0 

C_p(i)  =  C_p(i)  -  700; 

end 

l  Function  3  Penalty  Cost 
if  Sat_s{3}(s3,3)  ==  1 
C_p(i)  =  C_p(i)  +  0; 
elseif  Sat_s{3}(s3,3)  ==  0 
C_p(i)  =  C_p(i)  -  200; 

end 

end 


rarammmmrararammn; ramrararararararammm 


7,  Define  the  overall  constellation  action  set  in  a  cell  array 
Constellation_a  =  cell (actions'^) ;  al=l;  a2=l;  a3=l;  for  i  = 

1 : actions'^ 

Constellation_a{i}= [al  a2  a3] ; 

a3  =  a3  +  1 ; 
if  a3  >  actions 
a2  =  a2  +  1; 
a3  =  1; 
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end 


if  a2  >  actions 
al  =  al  +  1; 
a2  =  1; 

end 

end 


rarammmrarararamramrararammrammmmran 


7.  Define  the  overall  constellation  transition  probabilities.  Do  this 
%  by  looping  through  each  possibile  (s,a,j)  combination  in  the 
7,  constellation  and  multiplying  the  individual  satellite  transition 
7,  probabilities  that  would  lead  to  the  overall  satellite  constellation 
7,  transition.  The  user  of  this  code  must  know  clearly  what  state  the 
7,  index  of  each  satellite’s  state  and  action  vector  represents, 
si  =  1;  s2  =  1;  s3  =  1;  al  =  1;  a2  =  1;  a3  =  1;  jl  =  1;  j2  =  1;  j3 
=  1;  Constellation_p  =  zeros (TotStates ,actions~K, TotStates) ; 


for  s  =  l:TotStates 

for  a  =  1: actions'^ 

for  j  =  1: TotStates 

Constellation_p(s , a, j)  =  p{l}(sl,al, jl)*p{2}(s2,a2, j2) *p{3}(s3,a3, j3) 

7*  Eliminate  transition  probabilities  associated  with 
7,  infeasible  actions.  Set  the  transition  probabilities  to 
7,  0  for  all  infeasible  actions  and  therefore  sets  the 
l  expected  rewards  to  0  so  they  can  be  identified  and  then 
7,  set  to  NA. 
if  si  ==  1 

if  al  ==  2 

Constellation_p(s,a, j)  =  0; 

end 

end 

if  s2  ==  1 

if  a2  ==  2 

Constellation_p(s,a, j)  =  0; 

end 

end 

if  s3  ==  1 

if  a3  ==  2 

Constellation_p(s,a, j)  =  0; 

end 

end 
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°/*  The  previous  block  eliminates  any  on  orbit  repairs  to  a 

#/«  satellite  that  has  nothing  broken. 

if  si  ==  states(l) 

if  s2  ==  states (2) 

if  s3  ==  states (3) 
if  al  ==  1 

if  a2  ==  1 

if  a3  ==  1 

Constellation_p(s,a, j)  =  0; 

end 

end 

end 

end 

end 

end 

°/«  The  previous  block  eliminates  the  possibility  of  doing 

1  nothing  when  all  functions  of  all  satellites  are 

°/*  non-operational . 

j  3  =  j  3  +  1; 

if  j3  >  states(3) 
j3  =  1; 
j2  =  j2  +  1; 

end 

if  j2  >  states(2) 
j2  =  1; 
jl  =  jl  +  1; 

end 

if  jl  >  states(l) 
jl  =  1; 
a3  =  a3  +  1 ; 

end 

if  a3  >  actions 
a3  =  1; 
a2  =  a2  +  1; 

end 

if  a2  >  actions 
a2  =  1; 
al  =  al  +  1; 

end 

if  al  >  actions 
al  =  1; 
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s3  =  s3  +  1 ; 


end 

if  s3  >  states (3) 
s3  =  1; 
s2  =  s2  +  1; 

end 

if  s2  >  states(2) 
s2  =  1; 
si  =  si  +  1; 

end 

if  si  >  states(l) 
si  =  1; 

end 

end 

end 

end 


#/«  Sensitivity  analysis  portion  of  the  code 

mram%mrammmmrara%mmmrara%%ramramrara 


°/0  Begin  the  big  for  loop  that  perforins  the  sensitivity  analysis 
for  parameter  =  1:2 
for  base  =  1 : 100 
disp(base) 

°/0  On  orbit  repair  cost  by  state 
if  parameter  ==  1 

SpaceVehicle  =  -base*10; 

else 

SpaceVehicle  =  -450; 

end 

C_m  =  cell (K) ; 

C_m{l}  =  [-realmax  SpaceVehicle-15  SpaceVehicle-35  SpaceVehicle-50] ; 

C_m{2}  =  [-realmax  SpaceVehicle-15  SpaceVehicle-35  SpaceVehicle-50] ; 

C_m{3}  =  [-realmax  SpaceVehicle-20  SpaceVehicle-15  SpaceVehicle-35  SpaceVehicle-35  SpaceVehicle-55 
SpaceVehicle-50  SpaceVehicle-70] ; 

7.  Satellite  replacement  cost 
if  parameter  ==  2 

C_s  =  [-base*10  -base*10  (-base*10)-50] ; 

else 

C_s  =  [-500  -500  -520] ; 

end 
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disp(C_s) 
disp(C_m{l}(2) ) 


rararammmrararamy.rara%%ra%mmmmmmrara 


%  Define  the  Immediate  Rewards  for  each  individual  satellite 
l  ra(a,s)  ==>  Immediate  reward  when  choosing  action  a  while  in 
7.  state  s  (constellation)  . 

7,  Assume  that  the  repair  costs  are  the  same  for  either  function  on 
7,  all  satellites 

ra  =  zeros (actions'^,  TotStates) ; 

for  Cons_A  =  liactions^K 

for  Cons_S  =  l:TotStates 
for  k  =  1:K 

if  Constellation_a{Cons_A}(k)  ==  1 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  0; 

end 

if  Constellation_a{Cons_A}(k)  ==  2 

SatState  =  Constellation_s{Cons_S} (k) ; 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  C_m{k} (SatState) ; 

end 

if  Constellation_a{Cons_A}(k)  ==  3 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  C_s(k); 

end 

end 

end 

end 


7,  Define  the  Expected  Rewards 

7,  r(s,a)  ==>  The  expected  reward  when  in  state  s  and  choosing 
7,  action  a  for  the  constellation.  This  is  equivalent  to  the 
7.  immediate  reward  plus  the  expected  penalty  cost  over  the  next 
7,  period. 

r  =  zeros (TotStates, actions^K) ; 

for  s  =  l:TotStates 

for  a  =  1 : actions~K 

7,  Calculate  the  expected  penalty  cost  over  the  next  period 
7,  (the  sum  of  the  penalty  costs  times  the  probability  that 
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7.  that  cost  is  incurred)  . 

Expected_penalty  =  0; 

for  j  =  l:TotStates 

Expected_penalty  =  Expected_penalty  +  Constellation_p(s,a, j)*C_p(j) ; 

end 

7,  Assign  the  expected  reward  resulting  from  taking  action  a 
7,  in  state  s . 
if  Expected_penalty  ==  0 
r(s,a)  =  NA; 

else 

r(s,a)  =  ra(a,s)  +  Expected_penalty ; 

end 

end 

end 


7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7o7o7o7.7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o 


7,  Backward  Induction  Code 


7,  Define  the  u,  ustar  and  dstar  vector 
u  =  zeros (TotStates , actions ,N) ; 
ustar  =  zeros (TotStates ,N) ; 
dstar  =  zeros (TotStates ,N) ; 

7,  Note:  The  value  of  any  action  in  the  final  time  period  has  zero 
7.  reward  and  therefore  zero  utility 
7,  ustar (s,N)  =  0  for  all  s 

n  =  N  -  1; 
while  n  >=  1 

for  s  =  l:TotStates  7.  Loop  through  all  constellation  states 
ustar (s,n)  =  -realmax; 

for  a  =  l:actions~K  7.  Loop  through  all  actions 

if  r(s,a)  ~=  NA  7.  Infeasible  action  -  no  reward 

expected_value  =  0; 
for  j  =  IrTotStates 

expected_value  =  expected_value  +  Constellation_p(s,a, j)*ustar(j ,n+l) 
end  7.  End  For  Loop  for  expected  value 
u(s,a,n)  =  r(s,a)  +  expected_value ; 

if  u(s,a,n)  >  ustar(s,n) 
ustar(s,n)  =  u(s,a,n); 
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dstar(s,n)  =  a; 


end 


end 


end 


end 

n  =  n  -  1; 


end 


rararammmrararammmmrararamraramnran 


7.  Store  the  optimal  value  results  in  the  matrix  ’results’ 
results (1,  base)  =  base; 
for  index  =  l:TotStates 

results (index,  base)  =  ustar (index, 1) ; 

end 

7,  Store  the  optimal  policies  in  the  cell  array  "policies" 
policies{base}  =  dstar; 

clear  ustar; 
clear  dstar; 

end  7o  end  the  loop  that  controls  the  sensitivity  analysis 

7.  Determine  which  file  to  write  to 
if  parameter  ==  1 

fid  =  f open( ’Three_Sat_Repair_Cost . txt ’ , ’w’ ) ; 

else 

fid  =  f open ( ’Three_Sat_Replace_Cost .txt ’, ’w’ ) ; 

end 

7.  Write  the  results 
for  j=l:base 

fprintf(fid,  ’7«8.5f\t’,  j); 

end 

fprintf(fid,  ’\n’); 
for  i  =  l:TotStates; 
for  j=l:base 

fprintf  (f id,  ’7»8.5f\t  ’  , result s(i,j))  ; 

end 

fprintf (fid,  ’\n’); 

end 

for  j=l:base 

fprintf (fid,  ’\n  Parameter  Iteration  -  ’); 
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fprintf (fid,  ,#/0g\tJ,  j); 
fprintf (fid,  ,\n,)| 
for  s=l :TotStates 


for  n=l:N 

fprintf  (f id,  ’“/ogNt’,  policies{j} (s ,n) ) ; 

end 

fprintf (fid,  ’ \n’); 

end 

end 

°/«  Close  file 
f close(f id) ; 

end 
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Appendix  E.  One-Satellite  Two-Way  Analysis 


%7.7o7oy//o7.%y.7o%7.yoyo7o%%%7o7o%%7.7oyo%%7.7oy.%%7.yoyo7.%7//.7o%%%%7o7.%%7.7oyo7.%%7.yoyo%%%yo7o%%%7.7o7o%%%%7o%% 


7.  Author:  lLt  Tim  Cook 
7.  AFIT/ENS/G0R-05M 


7.  March  2005 

7,  This  program  performs  a  two-way  sensitivity  analysis  on  a  Markov  decision 
7.  process  formulated  to  find  an  optimal  maintenance  policy  for  a  one 
7,  satellite  constellation.  In  this  file,  the  parameters  that  are  modified 
7,  are  the  repair  costs  and  the  replacement  costs . 


7o7.7o7o7o7o7.7o7o7.7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7.7o7o7.7o7o7o7o7.7o7o7.7o7o7o7o7o7o7o7.7o 


7.  Define  Parameters 
N  =  20;  7.  Length  of  planning  horizon 
M  =  3;  7.  Number  of  functions 

7.  Total  number  of  states  that  the  satellite  can  possibly  enter  into 
TotStates  =  2~M; 

7.  Number  of  possible  actions 
actions  =  3; 

7.  Failure  rates  —  1/ (expected  lifetime  in  years)* (4  quarters) 
lambdal  =  1/(5. 5*4);  7.  Failure  rate  of  function  1 

lambda2  =  1/(5. 25*4);  7.  Failure  rate  of  function  2 

lambda3  =  1/(6. 5*4);  7.  Failure  rate  of  function  3 

7.  Penalty  costs 

C_p  =  [0  -200  -500  -600  -300  -400  -400  -700] ; 

7.  Allocate  memory  for  the  matrices  that  will  store  the  sensitivity 
7.  results  before  they  are  output  to  a  file 
results  =  zeros(20,  100);  policies  =  cell(2000); 

7.  Begin  the  outer  for  loop  that  varies  the  satellite  replacement  cost, 
for  C_L  =1:20 

7.  Satellite  replacement  cost 
C_s  =  -50*C_L; 

7.  Begin  the  inner  for  loop  that  varies  the  space  vehicle  cost  from  07. 
7.  to  1007.  of  the  satellite  replacement  cost, 
for  C_V  =  1:101 
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disp(  [C_L,C_V] ) 


°/0  On-orbit  repair  costs 
Launch  =  - ( (101-C_V) /100) *C_s ; 

C_m  =  [-realmax  -(Launch  +  20)  -(Launch  +  15)  -(Launch  +  35)  -(Launch  +  35)  -(Launch  +  55) 
-(Launch  +  50)  -(Launch  +  70)] ; 
disp( [C_s , Launch] ) 

°/0  Use  when  an  action  is  not  feasible  for  that  state 
NA  =  -99999; 


°/0  Failure  probabilities  for  each  function 
F  =  [1-exp (-lambdal)  1-exp (-lambda2)  1-exp (-lambda3)] ; 

°/0  Survival  probabilities  for  each  function 
R  =  [exp (-lambdal)  exp(-lambda2)  exp(-lambda3)] ; 

°/0  Probability  of  a  successful  repair  for  each  function 
M  =  [.95  .96  .97] ; 

°/0  Probability  of  a  successful  replacement  for  each  function 
G  =  [.975  .94  .98] ; 
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°/0  Define  Transition  Probabilities 

°/0  p(s,a,j)  ==>  Probability  of  going  to  state  j,  given  presently  in 
°/0  state  s  and  choosing  action  a 

p  =  zeros (TotStates ,  actions,  TotStates) ; 


p(l,l,l)  =  R(l) *R(2) *R(3) ; 
p(l , 1 , 2)  =  R(1)*R(2)*F(3); 
p(l , 1 , 3)  =  R(1)*F(2)*R(3) ; 
p(l , 1 ,4)  =  R(1)*F(2)*F(3) ; 
p(l , 1 , 5)  =  F(1)*R(2)*R(3); 
p(l , 1 , 6)  =  F(1)*R(2)*F(3); 
p(l , 1 , 7)  =  F(1)*F(2)*R(3); 
p(l , 1 , 8)  =  F(1)*F(2)*F(3) ; 


°/0  There  is  no  need  to  include  the  transition  probabilities  for  action  2 
°/0  while  in  state  2  because  the  action  will  never  be  chosen.  This  is  because 
•/.  the  transition  probabilities  are  identical  to  those  of  action  1,  but  the 
°/0  large  setup  cost  is  not  incurred  with  action  1. 


p(l ,3, 1)  =  G(l) *G(2) *G(3) ; 
p(l ,3,2)  =  G(1)*G(2)*(1-G(3)); 
p(l ,3,3)  =  G(1)*(1-G(2))*G(3); 
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p(l ,3,4) 
p(l ,3,5) 
p(l,3,6) 
p(l ,3,7) 
p(l ,3,8) 

p(2 , 1 ,2) 

p(2 , 1 ,4) 

p(2 , 1 ,6) 
p(2 , 1 ,8) 

p(2 ,2 , 1) 
p(2,2,2) 

p(2,2,3) 

p(2,2,4) 

p(2,2,5) 

p(2,2,6) 

p(2,2,7) 

p(2,2,8) 

p(2 ,3, 1) 
p(2,3,2) 
p(2,3,3) 
p(2,3,4) 
p(2,3,5) 
p(2,3,6) 
p(2,3,7) 
p(2,3,8) 

p(3, 1 ,3) 
p(3 , 1 ,4) 
p(3, 1 ,7) 
p(3 , 1 ,8) 

p(3 ,2 , 1) 
p(3,2,2) 
p(3,2,3) 
p(3,2,4) 
p(3,2,5) 
p(3,2,6) 
p(3,2,7) 
p(3,2,8) 


G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * ( 1— G (3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(l-G(l) ) * (1-G(2) ) *(1-G(3) ) ; 

R(1)*R(2) ; 

R(1)*F(2) ; 

F(1)*R(2); 

F(1)*F(2); 

R(1)*R(2)*M(3) ; 
R(1)*R(2)*(1-M(3)) ; 
R(1)*F(2)*M(3) ; 
R(1)*F(2)*(1-M(3)) ; 
F(1)*R(2)*M(3) ; 
F(1)*R(2)*(1-M(3)) ; 
F(1)*F(2)*M(3) ; 
F(1)*F(2)*(1-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(l-G(l) ) * (1-G(2) ) *(1-G(3) ) ; 

R(1)*R(3) ; 

R(1)*F(3) ; 

F(1)*R(3) ; 

F(1)*F(3) ; 

R(1)*M(2)*R(3) ; 
R(1)*M(2)*F(3) ; 
R(1)*(1-M(2))*R(3) ; 
R(1)*(1-M(2))*F(3) ; 
F(1)*M(2)*R(3) ; 
F(1)*M(2)*F(3) ; 
F(1)*(1-M(2))*R(3) ; 
F(1)*(1-M(2))*F(3) ; 
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p(3 ,3, 1) 
p(3,3,2) 
p(3,3,3) 
p(3,3,4) 
p(3,3,5) 
p(3,3,6) 
p(3,3,7) 
p(3,3,8) 

p(4,l,4) 

p(4,l,8) 

p(4,2 , 1) 
p(4,2,2) 
p(4,2,3) 
p(4,2,4) 
p(4,2,5) 
p(4,2,6) 
p(4,2,7) 
p(4,2,8) 

p(4,3,l) 

p(4,3,2) 

p(4,3,3) 

p(4,3,4) 

p(4,3,5) 

p(4,3,6) 

p(4,3,7) 

p(4,3,8) 

p(5,l,5) 
p(5,l,6) 
p(5 , 1 ,7) 
p(5,l,8) 

p(5 ,2 , 1) 
p(5,2,2) 
p(5,2,3) 
p(5,2,4) 
p(5,2,5) 
p(5,2,6) 
p(5,2,7) 
p(5,2,8) 


G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 

( 1— G ( 1 ) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 


R(l); 
F(l)  ; 


R(1)*M(2)*M(3) ; 
R(1)*M(2)*(1-M(3)) ; 
R(1)*(1-M(2))*M(3) ; 

R(l) * (1-M(2) ) * (1-M(3) ) ; 
F(1)*M(2)*M(3) ; 
F(1)*M(2)*(1-M(3)) ; 
F(1)*(1-M(2))*M(3) ; 
F(l)*(l-M(2))*(l-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) *(1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(l-G(l) ) * (1-G(2) ) *(1-G(3) ) ; 

R(2)*R(3) ; 

R(2)*F(3) ; 

F(2)*R(3) ; 

F (2)  *F  (3)  ; 

M(1)*R(2)*R(3) ; 
M(1)*R(2)*F(3) ; 
M(1)*F(2)*R(3) ; 
M(1)*F(2)*F(3) ; 
(1-M(1))*R(2)*R(3) ; 
(1-M(1))*R(2)*F(3) ; 
(1-M(1))*F(2)*R(3) ; 
(1-M(1))*F(2)*F(3) ; 


E-4 


p(5 ,3, 1) 
p(5,3,2) 
p(5,3,3) 
p(5,3,4) 
p(5,3,5) 
p(5,3,6) 
p(5,3,7) 
p(5,3,8) 

p(6,l,6) 

p(6,l,8) 

p(6 ,2 , 1) 
p(6,2,2) 

p(6,2,3) 

p(6,2,4) 

p(6,2,5) 

p(6,2,6) 

p(6,2,7) 

p(6,2,8) 

p(6,3,l) 

p(6,3,2) 

p(6,3,3) 

p(6,3,4) 

p(6,3,5) 

p(6,3,6) 

p(6,3,7) 

p(6,3,8) 

p(7, 1 ,7) 
p(7,l,8) 

p(7,2,l) 
p(7,2,2) 
p(7,2,3) 
p(7 ,2,4) 
p(7,2,5) 
p(7,2,6) 
p(7,2,7) 
p(7,2,8) 


G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

( 1— G ( 1 ) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 

(1-G (!))*(! -G(2))*(l-G (3)) ; 


R(2); 
F (2)  ; 


M(1)*R(2)*M(3) ; 
M(1)*R(2)*(1-M(3)) ; 
M(1)*F(2)*M(3) ; 
M(1)*F(2)*(1-M(3)) ; 
(1-M(1))*R(2)*M(3) ; 
(1-M(1))*R(2)*(1-M(3)) ; 
(1-M(1))*F(2)*M(3) ; 

(l-M(l) ) *F(2) * (1-M(3) ) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 

(1-G (!))*(! -G(2))*(l-G (3)) ; 


R(3)  ; 
F (3)  ; 


M(1)*M(2)*R(3) ; 
M(1)*M(2)*F(3) ; 
M(1)*(1-M(2))*R(3) ; 
M(1)*(1-M(2))*F(3) ; 
(1-M(1))*M(2)*R(3) ; 
(1-M(1))*M(2)*F(3) ; 
(1-M(1))*(1-M(2))*R(3) ; 
(1-M(1))*(1-M(2))*F(3) ; 
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p(7,3, 

,1) 

p(7,3. 

,2) 

p(7 ,3. 

,3) 

p(7,3. 

,4) 

p(7 ,3. 

,5) 

p(7 ,3. 

,6) 

p(7,3. 

,7) 

p(7 ,3. 

,8) 

p(8 , 1 . 

,8) 

p(8,2. 

,1) 

p(8,2. 

,2) 

p(8,2. 

,3) 

CO 

to 

,4) 

00 

to 

,5) 

00 

to 

,6) 

00 

to 

,7) 

CO 

to 

,8) 

p(8,3. 

,1) 

00 

00 

,2) 

V 

00 

00 

,3) 

CO 

00 

,4) 

p(8,3. 

,5) 

p(8,3. 

,6) 

00 

00 

,7) 

V 

00 

00 

,8) 

G ( 1) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 

G(l) *(1-G(2) ) * (1-G(3) ) ; 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

1; 

M(1)*M(2)*M(3) ; 
M(1)*M(2)*(1-M(3)) ; 
M(1)*(1-M(2))*M(3) ; 
M(l)*(l-M(2))*(l-M(3)) ; 
(1-M(1))*M(2)*M(3) ; 

(l-M(l) ) *M(2) * (1-M(3) ) ; 
(1-M(1))*(1-M(2))*M(3) ; 
(l-M(l) ) * (1-M(2) ) * (1-M(3) ) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 

(l-G(l) ) *G(2) * (1-G(3) ) ; 
(l-G(l) ) * (1-G(2) ) *G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 
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"/,  Define  the  Immediate  Rewards  -  deterministic  part  of  the  reward 
7  function  ra{a}(s)  ==>  Immediate  reward  when  choosing  action  a  in 
7  state  s 


ra  =  cell(actions) ; 

ra{l}  =  zeros (TotStates) 
ra{2}  =  zeros (TotStates) 
ra{3}  =  zeros (TotStates) 

ra{2}(l)  =  NA; 
ra{2}(2)  =  C_m(2) ; 
ra{2}(3)  =  C_m(3) ; 
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ra{2}(4)  =  C_m(4) ; 
ra{2}(5)  =  C_m(5) ; 
ra{2}(6)  =  C_m(6) ; 
ra{2>(7)  =  C_m(7) ; 
ra{2}(8)  =  C_m(8) ; 

ra{3}(l)  =  C_s; 
ra{3}(2)  =  C_s; 
ra{3>(3)  =  C_s; 
ra{3}(4)  =  C_s; 
ra{3}(5)  =  C_s; 
ra{3}(6)  =  C_s; 
ra{3}(7)  =  C_s; 
ra{3}(8)  =  C_s; 

°/0  Define  the  Expected  Rewards 

%  r(s,a)  ==>  The  expected  reward  when  in  state  s  and  choosing 
%  action  a.  This  is  equivalent  to  the  immediate  reward  plus  the 
%  expected  penalty  cost  over  the  next  period, 
r  =  zeros (TotStates , actions) ; 

for  s  =  1: TotStates 
for  a  =  1: actions 

/  Calculate  the  expected  penalty  cost  over  the  next  period 
7#  (the  sum  of  the  penalty  costs  times  the  probability  that 
°/0  that  cost  is  incurred)  . 

Expected_Penalty  =  0; 

for  j  =  1: TotStates 

Expected_Penalty  =  Expected_Penalty  +  p(s,a, j)*C_p(j) ; 

end 

7,  Assign  the  expected  reward  resulting  from  taking  action  a 
/  in  state  s . 
if  Expected_Penalty  ==  0 

r(s,a)  =  NA;  7,  The  action  is  not  feasible 

else 

r(s,a)  =  ra{a}(s)  +  Expected_Penalty ; 

end 

end 

end 
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l  Backward  Induction  Code  (Directly  adapted  from  Capt  Sumter’s 
7,  code) 


7,  Define  the  u,  ustar  and  dstar  vector 
u  =  zeros (TotStates , actions ,N) ; 
ustar  =  zeros (TotStates ,N) ; 
dstar  =  zeros (TotStates ,N) ; 

7,  Note:  The  value  of  any  action  in  the  final  time  period  has  zero 
7,  reward  and  therefore  zero  utility 
7,  ustar (s,N)  =  0  for  all  s 


n  =  N  -  1; 
while  n  >=  1 

for  s  =  1: TotStates 

ustar (s,n)  =  -realmax; 
for  a  =  1: actions 

if  r(s,a)  ~=  NA  7«  Infeasible  action  -  no  reward 
expected_value  =  0; 
for  j  =  IrTotStates 

expected_value  =  expected_value  +  p(s,a, j)*ustar(j ,n+l) 

end 

u(s,a,n)  =  r(s,a)  +  expected_value ; 


if  u(s,a,n)  >  ustar(s,n) 
ustar(s,n)  =  u(s,a,n); 
dstar(s,n)  =  a; 

end 

end 


end 

end 

n  =  n  -  1;  7«  Decrement  the  time 

end 
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7,  Store  the  optimal  value  results  (ustar)  in  the  matrix  ’results’ 
results(C_L,C_V)  =  ustar (1,1); 


7,  Store  the  optimal  policies  (dstar)  in  the  cell  array  "policies" 
policies{C_L,C_V}  =  dstar; 
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clear  ustar; 
clear  dstar; 

end  7,  end  the  repair  cost  loop 
end  7,  end  the  replacement  cost  loop 

7,  Open  file  to  write  results  to 
fid  =  f open( ’ One_Sat_Ratio . txt ’ , ’w’ ) ; 

7,  Write  the  results 
for  j=l : C_V 

fprintf (fid,  [num2str((101-j)/100) , *%’] ) ; 
end  fprintf (fid,  ’ \n’);  for  i  =  1:C_L; 
for  j=l : C_V 

fprintf  (fid,  ’7«8 . 5f  \t  ’  ,  results  (i ,  j  )  )  ; 

end 

fprintf (fid,  ’ \n’); 

end 

for  j=l:C_L 

fprintf (fid,  ’\n  Satellite  replacement  cost  -  ’); 
fprintf  (fid,  ’7og\t’,  50*j); 
fprintf (fid,  ’ \n’); 
for  i  =  1 : C_V 

fprintf (fid,  5 On-Orbit  cost  as  fraction  of  replacement  cost 
fprintf (fid,  [num2str ( (101-i) /100) ,*%*]); 
fprintf (fid,  ’ \n’); 
for  s=l :TotStates 
for  n=l:N 

fprintf  (fid,  ’  7og\t 5 ,  policies{j  ,  i}  (s  ,n)  )  ; 

end 

fprintf (fid,  ’Xn’); 

end 

end 

end 

7,  Close  file 
f close(f id) ; 
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Appendix  F.  Three- Satellite  Two-Way  Analysis 
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7.  Author:  lLt  Tim  Cook 
7.  AFIT/ENS/G0R-05M 


7.  March  2005 

7,  This  program  performs  a  two-way  sensitivity  analysis  on  a  Markov  decision 
7.  process  formulated  to  find  an  optimal  maintenance  policy  for  a  three 
7,  satellite  constellation.  In  this  file,  the  parameters  that  are  modified 
7,  are  the  repair  costs  and  the  replacement  costs . 
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7.  Define  Parameters 

N  =  20;  7.  Length  of  planning  horizon 

M  =  [2  2  3]  ;7.  Number  of  functions  desired  operational  in  each  satellite 

K  =  3;  7.  Number  of  Satellites 

7.  Create  a  vector  that  keeps  the  number  of  degradation  levels  for  each 

7.  satellite 

states  =  zeros (K);  for  i=l:K 

states(i)  =  2~M(i) ;  7.  Number  of  possible  states  for  satellite  i 

end 

7.  Define  the  total  number  of  states  that  the  entire  satellite 
7.  constellation  can  possibly  enter  into 
TotStates  =  prod(states ( : , 1) ) ; 

7.  Number  of  possible  actions  for  each  satellite 
actions  =  3; 

7.  Failure  rates  —  1/ (expected  lifetime  in  years)* (4  quarters) 
lambdal  =  1/(5. 5*4);  7.  Failure  rate  of  function  1 

lambda2  =  1/(5. 25*4);  7.  Failure  rate  of  function  2 

lambda3  =  1/(6. 5*4);  7.  Failure  rate  of  function  3 

7.  Allocate  memory  for  the  matrices  that  will  store  the  sensitivity 

7.  results  before  they  are  output  to  a  file 

results  =  zeros (TotStates ,  100);  policies  =  cell(100); 

7.  Use  when  an  action  is  not  feasible  for  that  state 
NA  =  -99999; 


7.  Failure  probabilities  for  each  function 
F  =  [1-exp (-lambdal)  1-exp (-lambda2)  1-exp (-lambda3)] ; 
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'/,  Survival  probabilities  for  each  function 
R  =  [exp(-lambdal)  exp(-lambda2)  exp(-lambda3)] ; 

"/,  Probability  of  a  successful  repair  for  each  function 
M  =  [.95  .96  .97] ; 

"/,  Probability  of  a  successful  replacement  for  each  function 
G  =  [.975  .94  .98] ; 


mmnnnmmmmnmmmmmnfflraxmmmnamxm 


"/,  Define  Transition  Probabilities 

1  p{k}(s,a,j)  ==>  Probability  for  satellite  k  of  going  to  state  j, 

"/,  given  presently  in  state  s  and  choosing  action  a  for 

'/,  that  individual  satellite 

'/,  Allocate  memory  for  the  transition  probability  matricies 

"/,  using  cell  arrays 

p  =  cell(K);  for  i  =  1:K 

p(i)  =  {zeros(states(i) ,  actions,  states(i))}; 

end 


"/,  Define  the  transition  probabilities 
p{l}(l,l,l)  =  R(l) *R(2) ; 
p{l} (1 , 1 , 2)  =  R(1)*F(2) ; 
p{l}(l,l,3)  =  F(1)*R(2); 
p{l} (1 , 1 ,4)  =  F(l) *F (2) ; 

"/,  There  is  no  need  to  include  the  transition  probabilities  for  action  2 
"/,  while  in  state  2  because  the  action  will  never  be  chosen.  This  is  because 
"/,  the  transition  probabilities  are  identical  to  those  of  action  1,  but  the 
"/,  large  setup  cost  is  not  incurred  with  action  1.  The  same  holds  for  the  other 
"/,  two  satellites. 


p{l}(l,3,l) 
p{l} (1 , 3 ,2) 
p{l} (1 , 3 ,3) 
p{l} (1 , 3 ,4) 


G(1)*G(2) ; 

G(l) *(1-G(2) ) ; 
(1-G(1))*G(2); 
(1-G(1))*(1-G(2)); 


p{l} (2 , 1 , 2)  =  R(l); 
p{l} (2,1,4)  =  F(l) ; 


p{l} (2 , 2 , 1) 
p{l}(2,2,2) 
p{l}(2,2,3) 
p{l}(2,2,4) 


R(1)*M(2) ; 
R(l)*(l-M(2))  ; 
F(1)*M(2) ; 

F(l) *(1-M(2) ) ; 
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p{l} (2 , 3 , 1)  =  G(1)*G(2); 
p{l>(2,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{l>(2,3,3)  =  ( 1— G ( 1) ) *G (2) ; 
p{l}(2,3,4)  =  (1-G(1))*(1-G(2)) ; 

p{l} (3 , 1 , 3)  =  R(2); 
p{l>(3,l,4)  =  F (2) ; 

p{l}(3 , 2 , 1)  =  M(1)*R(2); 
p{l>(3,2,2)  =  M(1)*F(2); 
p{l>(3,2,3)  =  (l-M(l) ) *R(2) ; 
p{l>(3,2,4)  =  (l-M(i))*F(2); 

p{l>(3,3,l)  =  G(1)*G(2) ; 
p{l>(3,3,2)  =  G(l) *(1-G(2) ) ; 
p{l>(3,3,3)  =  (1-G(1))*G(2); 
p{l>(3,3,4)  =  (1-G(1))*(1-G(2)) ; 

p{l}(4, 1 , 4)  =  1; 

p{l>(4,2,l)  =  M(1)*M(2); 
p{l>(4,2,2)  =  M(l)*(l-M(2)); 
p{l>(4,2,3)  =  (1-M(1))*M(2) ; 
p{l>(4,2,4)  =  (1-M(1))*(1-M(2)); 

p{l>(4,3,l)  =  G(1)*G(2) ; 
p{l}(4,3,2)  =  G(l)*(l-G(2)) ; 
p{l>(4,3,3)  =  ( 1— G ( 1) ) *G (2) ; 
p{l}(4,3,4)  =  (1-G(1))*(1-G(2)) ; 
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p{2}(l,l,l)  =  R(1)*R(2); 
p{2} (1 ,1,2)  =  R(l) *F (2) ; 
p{2} (1 ,1,3)  =  F(1)*R(2); 
p{2} (1 ,1,4)  =  F(1)*F(2); 

p{2>(l,3,l)  =  G(1)*G(2); 
p{2}(l,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{2}(l,3,3)  =  ( 1— G ( 1) ) *G (2) ; 
p{2} (1 ,3,4)  =  (1-G(1))*(1-G(2)); 

p{2} (2,1,2)  =  R(l) ; 
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p{2} (2,1,4)  =  F(l); 


p{2} (2,2,1)  =  R(1)*M(2); 
p{2} (2,2,2)  =  R(l)*(l-M(2)); 
p{2}(2,2,3)  =  F(1)*M(2); 
p{2}(2,2,4)  =  F(l)*(l-M(2)); 

p{2} (2,3,1)  =  G(1)*G(2) ; 
p{2}(2,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{2}(2,3,3)  =  (1-G(1))*G(2) ; 
p{2}(2,3,4)  =  (1-G(1))*(1-G(2)); 

p{2} (3,1,3)  =  R(2) ; 
p{2} (3,1,4)  =  F (2) ; 

p{2}(3,2,l)  =  M(1)*R(2); 
p{2}(3,2,2)  =  M(1)*F(2); 
p{2} (3,2,3)  =  (l-M(l) ) *R(2) ; 
p{2}(3,2,4)  =  (1-M(1))*F(2); 

p{2}(3,3,l)  =  G(1)*G(2) ; 
p{2}(3,3,2)  =  G(l)*(l-G(2)); 
p{2}(3,3,3)  =  (1-G(1))*G(2) ; 
p{2}(3,3,4)  =  (1-G(1))*(1-G(2)) ; 

p{2>(4,l,4)  =  1; 

p{2}(4,2,l)  =  M(1)*M(2); 
p{2}(4,2,2)  =  M(l)*(l-M(2)) ; 
p{2}(4,2,3)  =  (1-M(1))*M(2) ; 
p{2}(4,2,4)  =  (1-M(1))*(1-M(2)); 

p{2} (4,3,1)  =  G(1)*G(2) ; 
p{2}(4,3,2)  =  G ( 1 ) *(1-G(2) ) ; 
p{2}(4,3,3)  =  ( 1— G ( 1) ) *G (2) ; 
p{2}(4,3,4)  =  (1-G(1))*(1-G(2)) ; 
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p{3}(l,l,l)  =  R(1)*R(2)*R(3) ;  p{3}(l,l,2)  =  R(l) *R(2) *F(3) 
p{3>(l,l,3)  =  R(1)*F(2)*R(3) ;  p{3}(l,l,4)  =  R(l) *F(2) *F(3) 
p{3}(l,l,5)  =  F(1)*R(2)*R(3) ;  p{3>(l,l,6)  =  F(l) *R(2) *F(3) 
p{3>(l,l,7)  =  F(1)*F(2)*R(3) ;  p{3}(l,l,8)  =  F(l) *F(2) *F(3) 
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p{3}(l,3,l) 
p{3} (1 ,3,2) 
p{3>(l,3,3) 
p{3}(l,3,4) 
p{3>(l,3,5) 
p{3}(l,3,6) 
p{3} (1 ,3,7) 
p{3>(l,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(l-G(l) ) *G(2) *G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3); 
(1-G(1))*(1-G(2))*(1-G(3)); 


p{3}(2,l,2) 
p{3} (2,1,4) 
p{3} (2,1,6) 
p{3} (2,1,8) 


R(1)*R(2) ; 
R(1)*F(2) ; 
F(1)*R(2); 
F(1)*F(2); 


p{3>(2,2,l) 
p{3}(2,2,2) 
p{3} (2,2,3) 
p{3}(2,2,4) 
p{3}(2,2,5) 
p{3}(2,2,6) 
p{3}(2,2,7) 
p{3} (2,2,8) 


R(1)*R(2)*M(3) ; 
R(1)*R(2)*(1-M(3)) ; 
R(1)*F(2)*M(3) ; 
R(1)*F(2)*(1-M(3)) ; 
F(1)*R(2)*M(3) ; 
F(1)*R(2)*(1-M(3)) ; 
F(1)*F(2)*M(3) ; 
F(1)*F(2)*(1-M(3)) ; 


p{3}(2,3,l) 

p{3}(2,3,2) 

p{3}(2,3,3) 

p{3}(2,3,4) 

p{3}(2,3,5) 

p{3}(2,3,6) 

p{3}(2,3,7) 

p{3}(2,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(l))*(i-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


p{3} (3,1,3) 
p{3} (3,1,4) 
p{3} (3,1,7) 
p{3} (3,1,8) 


R(1)*R(3) ; 
R(1)*F(3) ; 
F(1)*R(3) ; 
F(1)*F(3); 


p{3} (3,2,1) 
p{3}(3,2,2) 
p{3}(3,2,3) 
p{3}(3,2,4) 
p{3}(3,2,5) 


R(1)*M(2)*R(3) ; 
R(1)*M(2)*F(3) ; 
R(1)*(1-M(2))*R(3) ; 
R(1)*(1-M(2))*F(3) ; 
F(1)*M(2)*R(3) ; 
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p{3}(3,2,6) 
p{3}(3,2,7) 
p{3} (3,2,8) 

p{3} (3,3,1) 
p{3}(3,3,2) 
p{3}(3,3,3) 
p{3} (3,3,4) 
p{3}(3,3,5) 
p{3}(3,3,6) 
p{3}(3,3,7) 
p{3}(3,3,8) 

p{3} (4,1,4) 
p{3} (4,1,8) 

p{3} (4,2,1) 
p{3}(4,2,2) 
p{3}(4,2,3) 
p{3}(4,2,4) 
p{3}(4,2,5) 
p{3}(4,2,6) 
p{3}(4,2,7) 
p{3}(4,2,8) 

p{3}(4,3,l) 

p{3}(4,3,2) 

p{3}(4,3,3) 

p{3}(4,3,4) 

p{3}(4,3,5) 

p{3}(4,3,6) 

p{3}(4,3,7) 

p{3}(4,3,8) 

p{3}(5,l,5) 
p{3} (5,1,6) 
p{3} (5,1,7) 
p{3} (5,1,8) 

p{3}(5,2,l) 

p{3}(5,2,2) 

p{3}(5,2,3) 

p{3}(5,2,4) 


F(l) *M(2) *F (3) ; 
F(1)*(1-M(2))*R(3) ; 
F(1)*(1-M(2))*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(l); 

F  ( 1 ) ; 


R(1)*M(2)*M(3) ; 
R(1)*M(2)*(1-M(3)) ; 
R(1)*(1-M(2))*M(3) ; 
R(l)*(l-M(2))*(l-M(3)) ; 
F(1)*M(2)*M(3) ; 
F(1)*M(2)*(1-M(3)) ; 
F(1)*(1-M(2))*M(3) ; 
F(l)*(l-M(2))*(l-M(3)) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)) ; 

R(2) *R(3) ; 

R(2) *F (3) ; 

F (2) *R(3) ; 

F (2) *F (3) ; 

M(l) *R(2) *R(3) ; 

M(l) *R(2) *F (3) ; 

M(l) *F (2) *R(3) ; 

M(l) *F (2) *F (3) ; 
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p{3}(5,2,5) 

p{3}(5,2,6) 

p{3}(5,2,7) 

p{3}(5,2,8) 

p{3>(5,3,l) 

p{3}(5,3,2) 

p{3}(5,3,3) 

p{3}(5,3,4) 

p{3}(5,3,5) 

p{3}(5,3,6) 

p{3}(5,3,7) 

p{3}(5,3,8) 

p{3} (6,1,6) 
p{3}(6, 1 ,8) 

p{3} (6,2,1) 
p{3}(6,2,2) 
p{3}(6,2,3) 
p{3}(6,2,4) 
p{3}(6,2,5) 
p{3}(6,2,6) 
p{3}(6,2,7) 
p{3}(6,2,8) 

p{3} (6,3,1) 
p{3}(6,3,2) 
p{3}(6,3,3) 
p{3}(6,3,4) 
p{3}(6,3,5) 
p{3}(6,3,6) 
p{3}(6,3,7) 
p{3}(6,3,8) 

p{3} (7,1,7) 
p{3} (7,1,8) 

p{3} (7,2,1) 
p{3}(7,2,2) 
p{3}(7 ,2,3) 
p{3}(7 ,2,4) 
p{3}(7 ,2,5) 


(1-M(1))*R(2)*R(3) ; 
(1-M(1))*R(2)*F(3) ; 
(1-M(1))*F(2)*R(3) ; 
(1-M(1))*F(2)*F(3) ; 

G(l) *G(2) *G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)) ; 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(2); 

F(2)  ; 


M(1)*R(2)*M(3) ; 
M(1)*R(2)*(1-M(3)) ; 
M(1)*F(2)*M(3) ; 
M(1)*F(2)*(1-M(3)) ; 
(1-M(1))*R(2)*M(3) ; 

(l-M(l) ) *R(2) * (1-M(3) ) ; 
(1-M(1))*F(2)*M(3) ; 
(1-M(1))*F(2)*(1-M(3)) ; 

G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(l))*(i-G(2))*G(3); 
(1-G(1))*(1-G(2))*(1-G(3)); 


R(3); 
F(3)  ; 


M(1)*M(2)*R(3) ; 
M(1)*M(2)*F(3) ; 
M(1)*(1-M(2))*R(3) 
M(1)*(1-M(2))*F(3) 
(1-M(1))*M(2)*R(3) 


F-7 


p{3}(7,2,6) 
p{3}(7,2,7) 
p{3}(7 ,2,8) 


(1-M(1))*M(2)*F(3) ; 
(1-M(1))*(1-M(2))*R(3) ; 
(1-M(1))*(1-M(2))*F(3) ; 


p{3} (7,3,1) 
p{3}(7 ,3,2) 
p{3}(7,3,3) 
p{3} (7,3,4) 
p{3}(7,3,5) 
p{3}(7,3,6) 
p{3}(7,3,7) 
p{3}(7,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3))  ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


p{3} (8,1,8)  =  1; 


p{3}(8,2,l) 

p{3}(8,2,2) 

p{3}(8,2,3) 

p{3}(8,2,4) 

p{3}(8,2,5) 

p{3}(8,2,6) 

p{3}(8,2,7) 

p{3}(8,2,8) 


M(1)*M(2)*M(3)  ; 
M(1)*M(2)*(1-M(3)) ; 
M(1)*(1-M(2))*M(3) ; 
M(l)*(l-M(2))*(l-M(3)) ; 
(1-M(1))*M(2)*M(3) ; 

(l-M(l) ) *M(2) * (1-M(3) ) ; 
(1-M(1))*(1-M(2))*M(3) ; 
(l-M(l) ) * (1-M(2) ) *(1-M(3) ) ; 


p{3}(8,3,l) 

p{3}(8,3,2) 

p{3}(8,3,3) 

p{3}(8,3,4) 

p{3}(8,3,5) 

p{3}(8,3,6) 

p{3}(8,3,7) 

p{3}(8,3,8) 


G(1)*G(2)*G(3) ; 
G(1)*G(2)*(1-G(3)) ; 
G(1)*(1-G(2))*G(3) ; 
G(l)*(l-G(2))*(l-G(3)); 
(1-G(1))*G(2)*G(3) ; 
(1-G(1))*G(2)*(1-G(3)); 
(1-G(1))*(1-G(2))*G(3) ; 
(1-G(1))*(1-G(2))*(1-G(3)); 


y:/.y.y:/:/.7.y:/:/.y.7.y:/:/.y.y:/:/.7.y:/:/.y.y:/:/:/.y.y:/:/.7.y:/:/.y.y:/;/.7.y.y:/;/.7.y:/.7.y.y.7:/:/.7.,/:/.7.y.y:/:/.7.,/:/.7.y.y:/:/.y. 


I  Define  the  state  space  for  each  satellite 
Sat_s  -  cell(K);  Sat_s{l}  =[11 


1  0 
0  1 


0  0]  ; 

Sat_s{2>  =[11 
1  0 
0  1 
0  0]  ; 
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Sat_s{3}  =[111 


110 
10  1 
10  0 
Oil 
0  10 
0  0  1 
0  0  0]; 


raramramrarararammmrararammrammmmnm 


7,  Define  the  overall  state  space  for  the  constellation 
Constellation_s  =  cell(TotStates) ;  si  =  1;  s2  =  1;  s3  =  1; 


for  i  =  l:TotStates 

Constellation_s{i}  =  [si  s2  s3] ; 

s3  =  s3  +  1 ; 

if  s3  >  states (3) 
s3  =  1; 
s2  =  s2  +  1; 

end 

if  s2  >  states (2) 
s2  =  1; 
si  =  si  +  1; 

end 

if  si  >  states(l) 
si  =  1; 

end 


end 


7o7.7o7o7o7oy.7o7oyo7oyo7o7oyo7o7o7o7o7o707o7o7o7o7o7oyo7o7o707o7.7o7oyo7o7o7o7oyo7o7o7o7o7.7o7o7o7o7o7o7o7o7.707oyo7o7o7o7o7.707oyo7o7o707o7o 


°/«  Penalty  Cost  assigned  per  state 
7,  The  decision  maker  desires  that 
C_p  =  zeros (TotStates) ;  for  i  =  l:TotStates 


°/«  Function  1  Penalty  Cost 
si  =  Constellation_s{i}(l) ; 
s2  =  Constellation_s{i}(2) ; 
s3  =  Constellation_s{i}(3) ; 

sum  =  Sat_s{l}(sl , 1)+Sat_s{2}(s2, 1)+Sat_s{3}(s3, 1) ; 
if  sum  ==  3 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  2 

C_p(i)  =  C_p(i)  +  0; 
elseif  sum  ==  1 
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C_p(i)  =  C_p(i)  -  500; 


elseif  sum  ==  0 

C_p(i)  =  C_p(i)  -  650; 

end 

°/«  Function  2  Penalty  Cost 

sum  =  Sat_s{l}(sl ,2)+Sat_s{2}(s2,2)+Sat_s{3}(s3,2) ; 
if  sum  ==  3 


C_p(i) 

=  C_p(i) 

+  0; 

elseif  sum 

==  2 

C_p(i) 

=  C_p(i) 

+  0; 

elseif  sum 

==  1 

C_p(i) 

=  C_p(i) 

-  450 

elseif  sum 

==  0 

C_p(i) 

=  C_p(i) 

-  700 

end 


l  Function  3  Penalty  Cost 
if  Sat_s{3}(s3,3)  ==  1 
C_p(i)  =  C_p(i)  +  0; 
elseif  Sat_s{3}(s3,3)  ==  0 
C_p(i)  =  C_p(i)  -  200; 

end 

end 


rararmrararammmra: ay.mny.mm 


7,  Define  the  overall  constellation  action  set  in  a  cell  array 
Constellation_a  =  cell (actions'^) ;  al=l;  a2=l;  a3=l; 


for  i  =  1: actions'^ 

Constellation_a{i}= [al  a2  a3]  ; 
a3  =  a3  +  1 ; 
if  a3  >  actions 
a2  =  a2  +  1; 
a3  =  1; 

end 

if  a2  >  actions 
al  =  al  +  1; 
a2  =  1; 

end 

end 


mmnnnmmmmnmmmmmnfflraiimmrannnmm 
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7#  Define  the  overall  constellation  transition  probabilities.  Do  this 

7,  by  looping  through  each  possibile  (s,a,j)  combination  in  the 

%  constellation  and  multiplying  the  individual  satellite  transition 

7,  probabilities  that  would  lead  to  the  overall  satellite  constellation 

7,  transition.  The  user  of  this  code  must  know  clearly  what  state  the 

7,  index  of  each  satellite’s  state  and  action  vector  represents. 

si  =  1; 

s2  =  1; 

s3  =  1; 

al  =  1; 

a2  =  1; 

a3  =  1; 

jl  =  1; 

j2  =  1; 

j3  =  1; 

Constellation_p  =  zeros (TotStates ,actions~K, TotStates) ; 

for  s  =  l:TotStates 

for  a  =  1: actions'^ 

for  j  =  1: TotStates 

Constellation_p(s ,a, j )  =  p{l}(sl ,al , j I)*p{2}(s2,a2, j2) *p{3}(s3,a3, j3) 

7o  Eliminate  transition  probabilities  associated  with 
l  infeasible  actions.  Set  the  transition  probabilities  to 
7,  0  for  all  infeasible  actions  and  therefore  sets  the 
7,  expected  rewards  to  0  so  they  can  be  identified  and  then 
7,  set  to  NA. 
if  si  ==  1 

if  al  ==  2 

Constellation_p(s ,a, j)  =  0; 

end 

end 

if  s2  ==  1 

if  a2  ==  2 

Constellation_p(s ,a, j)  =  0; 

end 

end 

if  s3  ==  1 

if  a3  ==  2 

Constellation_p(s ,a, j)  =  0; 

end 

end 

7»  The  previous  block  eliminates  any  on  orbit  repairs  to  a 
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°/*  satellite  that  has  nothing  broken. 


if  si  ==  states(l) 

if  s2  ==  states (2) 

if  s3  ==  states (3) 
if  al  ==  1 

if  a2  ==  1 

if  a3  ==  1 

Constellation_p(s,a, j)  =  0; 

end 

end 

end 

end 

end 

end 

1  The  previous  block  eliminates  the  possibility  of  doing 

°/0  nothing  when  all  functions  of  all  satellites  are 

l  non-operational . 

j3  =  j3  +  1; 

if  j3  >  states(3) 
j3  =  1; 
j2  =  j2  +  1; 

end 

if  j2  >  states(2) 
j2  =  1; 
jl  =  jl  +  1; 

end 

if  jl  >  states(l) 
jl  =  1; 
a3  =  a3  +  1 ; 

end 

if  a3  >  actions 
a3  =  1; 
a2  =  a2  +  1; 

end 

if  a2  >  actions 
a2  =  1; 
al  =  al  +  1; 

end 

if  al  >  actions 
al  =  1; 
s3  =  s3  +  1 ; 
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end 


if  s3  >  states (3) 
s3  =  1; 
s2  =  s2  +  1; 

end 

if  s2  >  states(2) 
s2  =  1; 
si  =  si  +  1; 

end 

if  si  >  states(l) 
si  =  1; 

end 


end 

end 

end 


rarammmraram%mrarararammmrammrararara 


7.  Sensitivity  analysis  portion  of  the  code 


mrammmrarararammmrararammmmmmnnra 


%  Begin  the  outer  for  loop  that  varies  the  satellite  unit  cost, 
for  C_L  =1:20 


l  Satellite  unit  cost 
C_unit  =  -50*C_L; 


%  Begin  the  inner  for  loop  that  varies  the  space  vehicle  cost  from  07, 
7.  to  1007,  of  the  satellite  unit  cost, 
for  C_V  =  1:101 

disp( [C_L,C_V]  ) 


7,  On-orbit  repair  costs 

SpaceVehicle  =  -((101-C_V)/100)*C_unit ;  7,  a  positive  number 
disp ( [C_unit , -SpaceVehicle] ) 


C_m  =  cell (K) ; 

C_m{l}  =  [-realmax  -(SpaceVehicle  +  15)  -(SpaceVehicle  +  35)  -(SpaceVehicle 

C_m{2}  =  [-realmax  -(SpaceVehicle  +  15)  -(SpaceVehicle  +  35)  -(SpaceVehicle 

C_m{3}  =  [-realmax  -(SpaceVehicle  +  20)  -(SpaceVehicle  +  15)  -(SpaceVehicle 

-(SpaceVehicle  +  55)  -(SpaceVehicle  +  50)  -(SpaceVehicle  +  70)]  ; 


50)]; 

50)]; 

35)  -(SpaceVehicle  +  35) 


7,  Satellite  replacement  cost 

C_s  =  [C_unit-30  C_unit-30  C_unit-50] ; 
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%  Define  the  Immediate  Rewards  for  each  individual  satellite 
l  ra(a,s)  ==>  Immediate  reward  when  choosing  action  a  while  in 
7,  state  s  (constellation)  . 

7,  Assume  that  the  repair  costs  are  the  same  for  either  function  on 
7,  all  satellites 

ra  =  zeros (actions~K,  TotStates); 

for  Cons_A  =  1: actions'^ 

for  Cons_S  =  l:TotStates 
for  k  =  1:K 

if  Constellation_a{Cons_A}(k)  ==  1 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  0; 

end 

if  Constellation_a{Cons_A}(k)  ==  2 

SatState  =  Constellation_s{Cons_S} (k) ; 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  C_m{k} (SatState) ; 

end 

if  Constellation_a{Cons_A}(k)  ==  3 

ra(Cons_A,Cons_S)  =  ra(Cons_A,Cons_S)  +  C_s(k); 

end 

end 

end 

end 

7,  Define  the  Expected  Rewards 

7,  r(s,a)  ==>  The  expected  reward  when  in  state  s  and  choosing 
7,  action  a  for  the  constellation.  This  is  equivalent  to  the 
7,  immediate  reward  plus  the  expected  penalty  cost  over  the  next 
7.  period. 

r  =  zeros (TotStates ,actions~K) ; 

for  s  =  l:TotStates 

for  a  =  1 : actions~K 

7,  Calculate  the  expected  penalty  cost  over  the  next  period 
7,  (the  sum  of  the  penalty  costs  times  the  probability  that 
7,  that  cost  is  incurred)  . 

Expected_penalty  =  0; 

for  j  =  1: TotStates 

Expected_penalty  =  Expected_penalty  +  Constellation_p(s ,a, j)*C_p(j) 

end 

7,  Assign  the  expected  reward  resulting  from  taking  action  a 
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7.  in  state  s . 
if  Expected_penalty  ==  0 
r(s,a)  =  NA; 

else 

r(s,a)  =  ra(a,s)  +  Expected_penalty ; 

end 

end 

end 


rararammmrararammmmrararamraramnran 


/  Backward  Induction  Code  (Directly  adapted  from  Capt  Sumter’s  code) 


%  Define  the  u,  ustar  and  dstar  vector 
u  =  zeros (TotStates , actions ,N) ; 
ustar  =  zeros (TotStates ,N) ; 
dstar  =  zeros (TotStates ,N) ; 

%  Note:  The  value  of  any  action  in  the  final  time  period  has  zero 
7.  reward  and  therefore  zero  utility 
7,  ustar (s,N)  =  0  for  all  s 


n  =  N  -  1; 
while  n  >=  1 

for  s  =  1:  TotStates  %  Loop  through  all  constellation  states 
ustar (s,n)  =  -realmax; 

for  a  =  l:actions~K  7*  Loop  through  all  actions 

if  r(s,a)  ~=  NA  7*  Infeasible  action  -  no  reward 

expected_value  =  0; 
for  j  =  IrTotStates 

expected_value  =  expected_value  +  Constellation_p(s,a, j)*ustar(j ,n+l) 
end  7«  End  For  Loop  for  expected  value 
u(s,a,n)  =  r(s,a)  +  expected_value ; 


if  u(s,a,n)  >  ustar(s,n) 


ustar (s ,n) 


u(s ,a,n) ; 


dstar(s,n)  =  a; 


end 


end 


end 


end 

n  =  n  -  1; 


end 


7o7.y////.7oyoy//o7.7oyo0/o%7.%yo7.%7//o7o7.%7//.7o7.0/o%7.7oyo0/o%7.yoy.7.%7.yoyo7.%7//.7o7o%%%7.7oy.0/o7o7.%y.7.%%7.7o7.% 
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%  Store  the  optimal  value  results  (ustar)  in  the  matrix  ’results 
results(C_L,C_V)  =  ustar(l,l); 

7.  Store  the  optimal  policies  (dstar)  in  the  cell  array  "policies 
policies{C_L,C_V}  =  dstar; 

clear  ustar; 
clear  dstar; 

end  7,  end  the  repair  cost  loop 
end  7«  end  the  replacement  cost  loop 

7,  Open  file  to  write  results  to 

fid  =  f open( ’Three_Sat_Ratio . txt 5 , ’w’ ) ; 

7,  Write  the  results 
for  j=l:C_V 

fprintf  (f  id,  [num2str (  (101-j ) /100)  ; 

end  fprintf (fid,  ’ \n’);  for  i  =  1:C_L; 
for  j=l : C_V 

fprintf  (fid,  57»8 . 5f  \t  ’  ,  results  (i ,  j  )  )  ; 

end 

fprintf (fid,  ’ \n’); 

end 

for  j=l : C_L 

fprintf (fid,  ’  \n  Satellite  replacement  cost  -  ’); 
fprintf  (fid,  ’70g\t’,  50*j); 
fprintf (fid,  ’ \n’); 
for  i  =  1:C_V 

fprintf (fid,  5 On-Orbit  cost  as  fraction  of  replacement  cost  -  ’) 
fprintf (fid,  [num2str((101-i)/100) ,*%*]); 
fprintf (fid,  5\n’); 
for  s=l :TotStates 
for  n=l:N 

fprintf  (fid,  J7og\t’,  policies{j  ,  i}  (s  ,n) ) ; 

end 

fprintf (fid,  ’\n’); 

end 

end 

end 

7.  Close  file 
f close(f id) ; 


F-16 


Bibliography 


1.  Advanced  Industrial  Science  and  Technology.  “AIST: 

Development  of  a  Space  Maintenance  Robot” . 

http://www.aist.go.jp/aist_e/latest_research/2002/20020523/20020523.html 

2.  Apgar,  H.,  D.  Bearden,  and  R.  Wong  (1999).  Cost  Modeling.  Space  Mission 
Analysis  and  Design.  3rd  ed.,  Microcosm  Press,  El  Segundo,  California  and 
Kluwer  Academic  Publishers,  Boston,  Massachusetts,  783-820. 

3.  Baker,  D.  (editor)  (2002-2003).  Jane’s  Space  Directory:  2002-2003.  Jane’s  Infor¬ 
mation  Group  Inc.  1340  Braddock  Place,  Suite  300,  Alexandria,  VA.  22314-1651. 

4.  Barlow,  R.  E.  and  F.  Proschan.  (1965)  Mathematical  Theory  of  Reliability.  John 
Wiley  &  Sons,  Inc.,  New  York. 

5.  Barlow,  R.  E.  and  L.  C.  Hunter.  (1960)  Optimum  preventive  maintenance  poli¬ 
cies.  Operations  Research.  8,  90-100. 

6.  Barlow,  R.  E.,  L.  C.  Hunter,  and  F.  Proschan.  (1963)  Optimum  checking  proce¬ 
dures.  Journal  of  the  Society  for  Industrial  and  Applied  Mathematics ,  4,  1078- 
1095. 

7.  Bloch-Mercier,  S.  (2002)  A  preventive  maintenance  policy  with  sequential  check¬ 
ing  procedure  for  a  Markov  deteriorating  system.  European  Journal  of  Operations 
Research ,  147,  548-576. 

8.  Brown,  G.,  R.  Dell,  H.  Holtz  and  A.  Newman.  (2003)  How  Space  Command 
optimizes  long  term  investment  in  space  systems.  Interfaces,  33,  (4),  1-14. 

9.  Campbell,  N.  R.  (1941).  The  replacement  of  perishable  members  of  a  continually 
operating  system.  J.  Roy.  Statist.  Soc.,  7,  110-130. 

10.  Clio,  I.D.,  and  Parlar,  M.  (1991)  A  survey  of  maintenance  models  for  multi-unit 
systems.  European  Journal  of  Operations  Research ,  51,  1-23. 

11.  Comstock,  J.  (2004).  Deputy  Program  Director,  Welkin  Associates,  Ltd.,  Chan¬ 
tilly,  VA.  Telephone  interview.  28  June  2004. 

12.  Ebcling,  C.E.  (1997)  An  Introduction  to  Reliability  and  Maintainability  Engi¬ 
neering.  McGraw-Hill  Companies,  Inc.,  Boston,  MA. 

13.  Gertsbakh,  I.  B.  (1984)  Optimal  group  preventive  maintenance  of  a  system  with 
observable  state  parameter.  Advances  in  Applied  Probability,  16,  923-925. 

14.  Grail,  A.,  C.  Berenguer,  and  L.  Dicullc.  (2002)  A  condition-based  maintenance 
policy  for  stochastically  deteriorating  systems.  Reliability  Engineering  and  Sys¬ 
tem  Safety,  76,  167-180. 


BIB-1 


15.  Giirler,  U.  and  A.  Kaya.  (2002)  A  maintenance  policy  for  a  system  with  multi¬ 
state  components:  an  approximate  solution.  Reliability  Enqineerinq  and  System 
Safety,  76,  117-127. 

16.  Hansen,  C.  K.  (1995).  Reliability  prediction  andsimulation  for  a 
communications-satellite  fleet.  1995  Proceedings,  Annual  Reliability  and  Main- 
tainibility  Symposium ,  Washington  D.C.,  Jan.  16-19,  152-158. 

17.  Jacobs,  J.  L.,  A.  J.  Amato,  M.  D.  Bolduc,  S.  P.  Butler,  and  J.  E.  Caspero 
(1992).  OSCARS  -  Operational  Constellation  Availability  and  Reliability  Sim¬ 
ulation.  Proceedings  of  the  1992  Summer  Computer  Simulation  Conference, 
Reno, Nevada,  July  27-30,  878-882. 

18.  Kececioglu,  D.  and  F.  Sun  (1995)  A  general  discrete-time  dynamic  programming 
model  for  the  opportunistic  replacement  policy  and  its  application  to  ball-bearing 
systems.  Reliability  Engineering  and  System  Safety,  47,  175-185. 

19.  Kulkarni,  V.  G.  (1995)  Modeling  and  Analysis  of  Stochastic  Systems.  Chapman 
&  Hall/CRC,  New  York. 

20.  Launius,  R.  D.  “Sputnik  and  the  Origins  of  the  Space  Age” 
http: //www. hq.nasa.gov/ofhce/pao/History/sputnik/sp  utorig.html 

21.  Lotka,  A.  J.  (1939)  A  contribution  to  the  theory  of  self-renewing  aggregates  with 
special  reference  to  industrial  replacement.  Annals  of  Mathematical  Statistics , 
10,  1-25. 

22.  Morse,  P.  M.  (1958)  Queues,  Inventories  and  Maintenance.  John  Wiley  &  Sons, 
Inc.,  New  York 

23.  Moustafa,  M.S.,  E.Y.  Abder  Maksoud,  S.  Sadek  (2004)  Optimal  major  and 
minimal  maintenance  policies  for  deteriorating  systems.  Reliability  Engineering 
and  System  Safety,  83,  363-368. 

24.  McCall,  J.  J.  (1965)  Maintenance  policies  for  stochastically  failing  equipment: 
a  survey.  Management  Science,  11,  493-524. 

25.  NASA  Goddard  Space  Flight  Center.  “The  Hubble  Project” 

http:  / /hubble. nasa.gov/ 

26.  Nummelin,  E.  (1980)  A  general  failure  model:  optimal  replacement  with  state 
dependent  replacement  and  failure  costs.  Annals  of  Mathematical  Statistics,  10, 
1-25. 

27.  On-Orbit  Servicing  (OOS).  “The  one  step  link  to  OOS”.  http://www. on-orbit¬ 
servicing,  com/ 

28.  Park,  K.  S.  (1988).  Optimal  wear-limit  replacement  with  wear-dependent  fail¬ 
ures.  IEEE  Transactions  on  Reliability,  37,  293-294. 


BIB-2 


29.  Pham,  H.,  and  H.  Wang  (2000).  Optimal  (t, T)  opportunistic  maintenance  of 
a  /e-out-of-n:G  system  with  imperfect  PM  and  partial  failure.  Naval  Research 
Logistics,  47,  223-239. 

30.  Pierskalla,  W.P.  and  J.A.  Voelker  (1976)  A  survey  of  maintenance  models: 
The  control  and  surveillance  of  deteriorating  systems.  Naval  Research  Logistics 
Quarterly,  23,  353-388. 

31.  Preinreich,  G.  A.  D.  (1940)  Economic  life  of  industrial  equipment.  Econometrica, 
8,  12-44. 

32.  Puterman,  M.  L.  (1994).  Markov  Decision  Processes:  Discrete  Stochastic  Dy¬ 
namic  Programming.  John  Wiley  &  Sons,  Inc.,  New  York. 

33.  Radner,  R,  and  D.  W.  Jorgenson  (1962).  Optimal  replacement  and  inspection  of 
stochastically  failing  equipment,  Chapter  12.  Studies  in  Applied  Probability  and 
Management  Science  Edited  by  Arrow,  Karlin,  and  Scarf.  Stanford  University 
Press,  Stanford,  California. 

34.  Radner,  R.,  and  D.  W.  Jorgenson  (1963).  Opportunistic  replacement  of  a  single 
part  in  the  presence  of  several  monitored  parts.  Managemerit  Science,  10,  70-84. 

35.  Reeves,  E.  I.  (1999).  Spacecraft  design  and  sizing.  Space  Mission  Analysis  and 
Design.  3rd  ed.,  Microcosm  Press,  El  Segundo,  California  and  Kluwer  Academic 
Publishers,  Boston,  Massachusetts,  301-352. 

36.  Sherif,  Y.  S.  and  M.  L.  Smith  (1981)  Optimal  maintenance  models  for  systems 
subject  to  failure:  A  review  Naval  Research  Logistics  Quarterly,  28,  47-74. 

37.  Sparte,  Inc.  “SEAS  -  System  Effectiveness  Analysis  Simulation” . 
http://www.teamseas.com/  14  Jan  2005. 

38.  Su,  C-T,  S-C  Wu,  and  C-C  Chang  (2000).  Multiaction  maintenance  subject  to 
action-dependent  risk  and  stochastic  failure.  European  Journal  of  Operations 
Research,  125,  133-148. 

39.  Sumter,  B.  R.  Optimal  Replacement  Policies  for  Satellite  Constellations.  MS 
Thesis,  AFIT/GOR/ENS/02-23.  Graduate  School  of  Engineering  and  Manage¬ 
ment,  Air  Force  Institute  of  Technology  (AU),  Wright-Patterson  AFB,  OH, 
March  2003 

40.  TheStreet.Com.  “AT&T  Wireless  Posts  Surprise  Profit”. 

http://www.thestreet.com/_tscs/tech/tclecom/10172498.html  21  July  2004 

41.  United  States  Air  Force.  “Air  Force  Link”  http://www.af.mil/factsheets/ 

42.  Valdez-Flores,  C.  and  R.  M.  Feldman  (1989).  A  survey  of  preventive  mainte¬ 
nance  models  for  stochastically  deteriorating  single-unit  systems.  Naval  Research 
Logistics,  36,  419-446. 


BIB-3 


43.  Vergin,  R.  C.,  and  M.  Scriabin  (1977).  Maintenance  scheduling  for  multicompo¬ 
nent  equipment.  AIIE,  9,  297-305. 

44.  Wang,  H.  (2002)  A  survey  of  maintenance  policies  of  deteriorating  systems. 
European  Journal  of  Operations  Research ,  139,  469-489. 

45.  Wilson,  S.  T.  (1990)  Measuring  the  Effectiveness  of  Space:  Satellite  Weather 
Systems.  M.S.  Thesis,  Graduate  School  of  Engineering  and  Management,  Air 
Force  Institute  of  Technology,  Wright-Patterson  AFB,  Ohio. 


BIB-4 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  074-0188 

The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  the  collection  of  information,  including 
suggestions  for  reducing  this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway, 

Suite  1204,  Arlington,  VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  an  penalty  for  failing  to  comply  with  a  collection  of 
information  if  it  does  not  display  a  currently  valid  OMB  control  number. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DD-MM-YYYY) 
01-03-2005 

2.  REPORT  TYPE 

Master’s  Thesis 

3.  DATES  COVERED  (From  -  To) 

Mar  2004  -  Mar  2005 

4.  TITLE  AND  SUBTITLE 

Optimal  Maintenance  for  Stochastically  Degrading  Satellite  Constellations 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

Cook,  Timothy,  J.,  First  Lieutenant,  USAF 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAMES(S)  AND  ADDRESS(S) 

Air  Force  Institute  of  Technology 

Graduate  School  of  Engineering  and  Management  (AFIT/EN) 

2950  Hobson  Street,  Building  642 

WPAFB  OH  45433-7765 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

AFIT/GOR/ENS/05-05 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

National  Reconnaissance  Office 

Attn:  William  J.  Comstock,  Rm  43D19H 

14675  Leed  Road 

Chantilly,  VA  20151-1715 
(703)  808-4436 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

This  thesis  develops  a  methodology  to  determine  an  optimal  policy  for  maintaining  a  satellite  constellation  that  degrades  over  time. 
Previous  work  has  developed  a  methodology  to  compute  an  optimal  replacement  policy  for  a  satellite  constellation  in  which  satellites  were  viewed 
as  binary  entities,  either  operational  or  failed.  This  research  extends  the  previous  models  by  developing  an  optimal  maintenance  policy  for  satellite 
constellations  in  which  each  satellite  may  operate  in  a  finite  number  of  degraded  states.  The  constellation  is  assumed  to  consist  of  a  finite  number  of 
satellites,  each  with  a  finite  number  of  functions  with  distinct  failure  mechanisms.  Assuming  each  function  lifetime  is 

exponentially  distributed,  the  stochastic  degradation  process  is  modelled  as  a  discrete-time  Markov  chain.  The  degradation  process  is  subsequently 
used  to  formulate  an  optimization  problem  as  a  finite  planning  horizon  Markov  decision  process  in  which  the  total  expected  loss  of  utility  is 
minimized.  The  maintenance  actions  considered  include  on-orbit  repairs  and  satellite  replacements,  each  of  which  has  an  associated  level  of  risk. 
Numerical  examples  are  presented  to  illustrate  the  model,  and  a  parametric  sensitivity  analysis  is  perfonned  using  notional  data  to  determine 
conditions  for  which  the  resulting  policy  consists  of  only  replacements,  only  on-orbit  repairs,  or  mixtures  of  replacements 
and  on-orbit  repairs. 

15.  SUBJECT  TERMS 

Satellite  Constellations,  Optimal  Maintenance  Policy,  Markov  Chains,  Dynamic  Programming,  Stochastic  Degradation 


1  16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 

18.  NUMBER 

19a.  NAME  OF  RESPONSIBLE  PERSON 

ABSTRACT 

OF 

Jeffrey  P.  Kharoufeh,  Phd,  (ENS) 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

PAGES 

19b.  TELEPHONE  NUMBER  (Include  area  code) 

u 

u 

u 

UU 

189 

(937)  785-3636  ext  4603;  e-mail:  Jeffrey.Kharoufeh@afit.edu 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39-18 


